Computer-readable recording medium having template for easy preparing multimedia contests

ABSTRACT

A computer-readable recording medium recording multimedia contents data and a data structure that represents a chronological sequence of operations that are processed in a data processing device. The computer-readable recording medium includes a schedule portion including a predetermined reproduction time when the media data is to be reproduced in the reproducing unit and a predetermined input time when the input operation is to be received by the input unit. The data processing device performs steps of processing the multimedia contents data so that the predetermined media data is reproduced at the predetermined reproduction time on the basis of a schedule portion, receiving a predetermined input operation at the predetermined input time. The data processing device carries out a predetermined editing operation and resumes processing of the multimedia contents in accordance with the schedule portion.

TECHNICAL FIELD

The present invention relates to a multimedia data processing device, amultimedia data processing program product and a recording mediumrecording multimedia contents data, and in particular, to a multimediadata processing device which can easily prepare multimedia contents, amultimedia data processing program product and a recording mediumrecording multimedia contents data.

BACKGROUND ART

Along with the spread of communication apparatuses such as cellularphones in recent years, communications where prepared multimediacontents are transmitted to other users or have multimedia contentsprepared for other users are spreading.

When multimedia contents are prepared, however, in many cases, complexoperation procedures are required, and in some cases, such preparationis difficult for users who are not accustomed to the operation.

Therefore, a technique aiming to simplify user operation has beenconsidered.

As such a communication tool, Japanese Patent Laying-Open No.2002-140270 discloses a receiving device which samples and carries outan attached script, interlocked with the opening of an e-mail, uponreception of an e-mail to which the script has been attached, andfurthermore, a receiving device which prepares a script that reflectsthe action that has been carried out in the receiving device and whichcontrols the output of the processing result in a manner where it isoutputted through the transmitter of the e-mail. The action that isdesired by the transmitting side can be carried out by the receivingside by using such a receiving device as that described in JapanesePatent Laying-Open No. 2002-140270.

Furthermore, Japanese Patent Laying-Open No. 2002-324036 describes atechnique for preparing an e-mail template where the layout informationof the contents is described in order to easily prepare an e-mail towhich multimedia data is to be attached, and for preparing en e-mailthat includes multimedia in accordance with the layout information ofthis template.

In the case where the above receiving device described in JapanesePatent Laying-Open No. 2002-140270 is made to carry out preparation of ascript, and thereby, multimedia contents are prepared, a problem ariseswhere it is necessary to describe a complex script and to transmit thisto the receiving device. In addition, a problem arises where in somecases, the script that is carried out does not induce an operation onthe receiving side, and therefore, the action desired by thetransmitting side is not necessarily obtained.

Furthermore, there is a problem where it is not easy to prepare ane-mail on the message preparing side of an e-mail or the like. That isto say, it is difficult to prepare a message where a script and datathat is prepared by a message preparer are simply combined.

The technology which is disclosed in Japanese Patent Laying-Open No.2002-324036 as a technology for facilitating preparation of a message oran e-mail that includes multimedia data and which facilitatesarrangement of letters, still pictures and the like by using a templateon which layout information is described works effectively in the casewhere an e-mail where a number of images and texts are mixed on onelarge screen which is viewed on a PC (Personal Computer) is prepared.However, the Patent Document does not state a case where an e-mail wherethe e-mail screen changes chronographically and discontinuously, andthus, the contents are switched in such a manner as scene A, scene B andscene C is prepared, and a problem arises where it is difficult for theuser to prepare multimedia data that transitions chronographically, evenin the case where the technology disclosed in the Patent Document isused.

Furthermore, a problem arises where the contents of the message cannotbe rewritten by utilizing a variety of functions, such as camera,microphone, and address list, which are held inherently by a terminal ina cellular phone or in a portable terminal that is a conventional dataprocessing device.

Furthermore, a problem arises where, in a terminal on the preparing sidethat is a conventional data processing device, a portion of a scriptthat is utilized for the preparation of multimedia contents cannot berewritten, and preparation and reproduction cannot be carried out byrewriting a minimal portion of the script when such a script is utilizedon the reproducing side.

Furthermore, a problem arises where, in a terminal on the preparing sidethat is a conventional data processing device, script commands or tagswhich are utilized for the preparation of multimedia contents, and whichare not required by the receivers or do not follow the standardprocedure cannot be eliminated.

The present invention is provided in order to solve such problems, and afirst object of the invention is to provide a multimedia data processingdevice which can easily prepare multimedia contents, a multimedia dataprocessing program and a data structure for multimedia contents data,and furthermore, a second object of the invention is to provide amultimedia data processing device that allows a receiver to carry outthe operation of preparing and editing desired multimedia contentswithout fail, a multimedia processing program and a data structure formultimedia contents data.

DISCLOSURE OF THE INVENTION

The present invention provides a multimedia data processing device asshown in the following, in order to solve the above-described problems.That is to say, in order to achieve the above-described objects, amultimedia data processing device according to one aspect of the presentinvention is provided with a script implementing unit for reproducingdata that is referred from a script that is included in a template inaccordance with the script, a user changeable data determining unit fordetermining a portion of data that is referred from the script and thatcan be changed by the user, a data changing unit for changing theportion that can be changed, and a script and data outputting unit foroutputting at least either the script or the changed data.

A multimedia data processing device according to another aspect of thepresent invention is provided with a script implementing unit forreproducing data that is referred from a script that is included in atemplate in accordance with the script, a function selecting unit forselecting a function of the multimedia data processing device inaccordance with the script that is included in the template, an actiondetermining unit for determining which process is to be carried out onthe data that is referred from the script by using the selected functionin accordance with the script that is included in the template, anaction implementing unit for implementing the determined process, a datachanging unit for changing the data that is referred from the script,and a script and data outputting unit for outputting at least either thescript or the changed data.

A multimedia data processing device according to still another aspect ofthe present invention is provided with a template list acquiring unitfor acquiring a list of templates that have been formed so as to includescripts where a predetermined action that requires to be carried out byanother device on the receiving side and time information for when theaction is required are described, a template list displaying unit fordisplaying a list of templates, a template selecting unit for selectingone or more desired template from a list of displayed templates, and amessage transmitting unit for transmitting the selected template toanother device as a message.

A multimedia data processing device according to yet another aspect ofthe present invention is provided with a template acquiring unit foracquiring a template that has been formed so as to include a script, aprogram acquiring unit for acquiring a program from a broadcastingstation, a contents acquiring unit for acquiring internet contents whichare provided via the internet, and a script implementing unit forreproducing at least one of data that is referred from the script, aprogram that has been acquired by the program acquiring unit, andinternet contents that have been acquired by the contents acquiring unitin accordance with the script that is included in the template in aposition on the screen and according to timing which are designated bythe script.

A multimedia data processing program product according to still yetanother aspect of the present invention is a program product forallowing a computer to carry out the process of a template that has beenformed so as to include a script which allows the computer to carry outthe script implementing step of reproducing data that is referred from ascript that is included in the template in accordance with the script,the user changeable data determining step of determining a portion thatcan be changed by a user from among data that is referred from thescript, the data changing step of changing the changeable portion byusing a function of a multimedia data processing device, and the scriptand data outputting step of outputting at least one of the script andthe changed data.

A multimedia data processing program product according to another aspectof the present invention is a program product for allowing a computer tocarry out the process of a template that has been formed so as toinclude a script which allows the computer to carry out the scriptimplementing step of reproducing data that is referred from a scriptthat is included in the template in accordance with the script, theterminal function selecting step of selecting a function of a multimediadata processing device in accordance with the script that is included inthe template, the action determining step of determining which processis to be carried out for the data that is referred from the script byusing the selected function, the action implementing step ofimplementing the determined process, the data changing step of changingthe data that is referred from the script, and the script and dataoutputting step of outputting at least one of the script and the changeddata.

A recording medium according to still another aspect of the presentinvention is a recording medium recording multimedia contents datahaving a data structure which is processed in a data processing devicethat includes a reproducing means for reproducing media data and aninputting means for receiving an input operation from the user, andwhich includes a reproduction describing unit for showing media datathat is reproduced in the reproducing means of the data processingdevice and an input operation describing unit for showing an inputoperation that is received by the inputting means of the data processingdevice and a process that corresponds to the input operation, whereinthe multimedia contents data is processed so that predetermined mediadata is reproduced in the reproducing means of the data processingdevice, a predetermined input operation is received by the inputtingmeans of the data processing device, and the data processing devicecarries out a process in accordance with the predetermined inputoperation on the basis of the input operation describing unit.

A data structure for multimedia contents data according to yet anotheraspect of the present invention is a data structure for multimediacontents data which is processed in a data processing device thatincludes a reproducing unit for reproducing media data and an inputtingunit for receiving an input operation from the user, and which includesa reproduction describing unit for showing media data that is reproducedin the reproducing unit of the data processing device, an inputoperation describing unit for showing an input operation that isreceived by the inputting unit of the data processing device, and aprocess that corresponds to the input operation, and a scheduledescribing unit for managing time of effect of the media data that isreproduced in the reproducing unit of the data processing device andtime of effect of the input operation that is received by the inputunit, wherein the multimedia contents data is processed so thatpredetermined media data is reproduced in the reproducing unit of thedata processing device at a predetermined time on the basis of theschedule describing unit, a predetermined input operation is received bythe inputting unit of the data processing device at a predeterminedtime, and the data processing device carries out a process thatcorresponds to the predetermined input operation at the predeterminedtime on the basis of the input operation describing unit.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing the configuration of a data communicationsystem according to the present embodiment.

FIG. 2 is a block diagram showing a concrete example of the hardwareconfiguration of a cellular phone 1 which is a reproducing and editingdevice that is included in the data communication system.

FIG. 3 is a diagram showing the programs which are stored in memory unit130 of cellular phone 1 and the flow of data processing in cellularphone 1.

FIG. 4 is a block diagram showing the functional configuration of areproducing and editing engine program 131.

FIG. 5 is a diagram schematically showing the data structure of ascript.

FIGS. 6, 7 and 47 are diagrams showing concrete examples of scripts.

FIG. 8 is a diagram showing a concrete example of a table thatprescribes the correspondence between startup applications and UI'swhich are displayed at the time of startup.

FIG. 9 is a flowchart showing reproducing and editing processing of ascript in cellular phone 1.

FIG. 10 is diagram showing a concrete example of a time table of mediadata drawing processing.

FIG. 11 is a diagram showing a concrete example of an event processingtime table.

FIG. 12 is a diagram showing a concrete example of a UI allocation statedisplaying time table.

FIG. 13 is a flowchart showing a process in accordance with the eventthat is implemented in step S18.

FIG. 14 is a flowchart showing a determining process of the processcontents in step S24.

FIG. 15 is a flowchart showing a menu process in step S25.

FIG. 16 is a diagram showing a concrete example of a menu screen.

FIG. 17 is a flowchart showing a link transfer process in step S27.

FIG. 18 is a flowchart showing an application implementing process instep S26.

FIGS. 19 and 20 are flowcharts showing processes at the time when lines20 and 23 of the script shown in FIG. 7 are implemented.

FIGS. 21, 22, 33 and 85 are diagrams showing concrete examples oftransitions of display screens at the time when a script is implemented.

FIG. 23 is a flowchart showing a preview process of a script in cellularphone 1.

FIG. 24 is a flowchart showing an application startup process in stepS62.

FIG. 25 is a flowchart showing a data acquiring process in step S64.

FIG. 26 is a flowchart showing a script rewriting process in step S66.

FIG. 27 is a flowchart showing an event time table referring process instep S33.

FIG. 28 is a diagram showing a concrete example of a table thatprescribes correspondence between logic symbols and hardware keys.

FIGS. 29 and 30 are diagrams showing concrete examples of scripts.

FIGS. 31 and 32 are flowcharts showing processes at the time when lines19 and 21 of the script shown in FIG. 30 are implemented.

FIG. 34 is a flowchart showing an e-mail attaching process of a scriptin cellular phone 1.

FIG. 35 is a flowchart showing a slide eliminating process of a scriptin cellular phone 1.

FIG. 36 is a diagram showing a concrete example of a menu screen at thetime of the slide eliminating process.

FIG. 37 is a flowchart showing slide rewriting prohibition process of ascript in cellular phone 1.

FIG. 38 is a flowchart showing an event process in step S96.

FIG. 39 is a diagram showing a concrete example of a menu screen at thetime of the preview process.

FIG. 40 is a diagram showing a concrete example of a screen transitionat the time of the preview process.

FIG. 41A is a diagram showing a concrete example of display screensbefore template rewriting.

FIG. 41B is a diagram showing a concrete example of display screensafter template rewriting.

FIG. 42 is a block diagram showing the functional configuration ofreproducing and editing engine program 131 according to the secondembodiment.

FIG. 43 is a flowchart showing a process in the reproducing and editingengine program.

FIG. 44 is a flowchart showing a data outputting process in step S1201.

FIG. 45 is a diagram showing a concrete example of a screen transitionof rewriting by means of a camera startup application.

FIG. 46 is a flowchart showing a reproducing and editing process in stepS1101.

FIG. 48 is a block diagram showing the functional configuration of areproducing and editing engine program 131 according to a modificationof the second embodiment.

FIG. 49 is a diagram schematically showing that a different function isselected even when the script is the same in accordance with a terminal.

FIGS. 50 and 61 are diagrams showing concrete examples of scripts whichare utilized in a terminal on the preparing side.

FIGS. 51A to 51D are diagrams showing concrete examples of thumbnaildisplays for each scene.

FIG. 52 is a diagram showing a concrete example of screen transition ofrewriting by means of an inline camera application and an inline textinputting application.

FIG. 53 is a diagram showing a concrete example of screen transition ofrewriting by means of a file acquiring application.

FIGS. 54A and 54B are diagrams showing concrete examples of screens forselection from among a number of processes which accept a selection of aprocess that is to be started from the present time.

FIG. 55 is a diagram showing a concrete example of a table that holdscorrespondence between transmitted messages and templates that have beenutilized for preparing messages.

FIGS. 56A to 56C are diagrams showing an example where media data andacquired data which are referred from reference information of a scriptare synthesized at a point in time of a time parameter value of anevent.

FIGS. 57A and 57B are diagrams showing an example of writing theacquired data into points in time of time parameter values of the event.

FIG. 58 is a diagram showing a concrete example of a switching table.

FIG. 59 is a flowchart showing determining process in step S1110.

FIG. 60 is a diagram showing a concrete example of a table which recordscorrespondence between the data acquisition programs which are stored incellular phone 1 and the media types supported by cellular phone 1.

FIG. 62 is a flowchart showing a data outputting process in step 1201.

FIG. 63 is a diagram showing a concrete example of an action managingtable.

FIG. 64 is a diagram showing a concrete example of a scene event table.

FIG. 65 is a flowchart showing a changing process in step S1205.

FIG. 66 is a flowchart showing an image taking application startupprocess in step S102.

FIG. 67 is a flowchart showing a process for determining the point intime when script reproduction is resumed.

FIG. 68 is a diagram showing a concrete example of a script wherewhether slide editing is permitted or prohibited is described.

FIG. 69 is a flowchart showing a process for determining permission forimplementing slide editing in step S42.

FIG. 70 is a diagram showing a concrete example of a script thatincludes a drawing information describing portion in the case where thescript is formed so as to be divided into the drawing informationdescribing portion and a user navigation describing portion.

FIG. 71 is a diagram showing a concrete example of a script thatincludes a user navigation describing portion in the case where thescript is formed so as to be divided into the drawing informationdescribing portion and a user navigation describing portion.

FIGS. 72A to 72D are diagrams showing a concrete example of screentransition at the time when a script is implemented.

FIG. 73 is a diagram showing a concrete example of a script thatincludes description which relates two scripts to each other.

FIG. 74 is a diagram showing the configuration of a data communicationsystem that is a formed so as to include a television set which is amultimedia data processing device according to the third embodiment.

FIG. 75 is a diagram showing a concrete example of a script thatincludes a drawing information describing portion that describes drawinginformation according to the third embodiment.

FIG. 76 is a diagram showing a concrete example of a script thatincludes a program table information describing portion that describesprogram information in accordance with time slots according to the thirdembodiment.

FIG. 77 is a diagram showing a concrete example of a script thatincludes a description which relates two scripts to each other.

FIG. 78 is a flowchart showing the flow of a process for determining thecontents to be received and reproduced.

FIG. 79 is a flowchart showing the flow of reproducing and editingprocess in the case where a script is described with SMIL.

FIGS. 80A and 80B are diagrams showing a concrete example of the datastructure that stores the results of analysis of a script.

FIG. 81 is a diagram showing a concrete example of the results ofanalysis of a script.

FIG. 82 is a diagram showing a concrete example of the results ofanalysis of an updated script.

FIG. 83 is a diagram showing a concrete example of a script on which arewriting process for adding media data has been carried out.

FIG. 84 is a diagram showing a concrete example of a script from whichthe editing function has been eliminated after a rewriting process foradding media data has been carried out.

BEST MODES FOR CARRYING OUT THE INVENTION

In the following, the preferred embodiments of the present invention aredescribed with reference to the drawings. In the following description,the same symbols are attached to parts and components that are the same.The names and functions of these are also the same. Accordingly,detailed descriptions thereof are not repeated.

First Embodiment

With reference to FIG. 1, a data communication system according to thepresent embodiment is formed so as to include a template providingserver (hereinafter simply referred to as server) 2 for providing atemplate that is described in a script language, and a reproducing andediting devices 1 which are multimedia data processing devices forreproducing and editing a provided template, wherein server 2 andreproducing and editing devices 1 communicate via the internet, a publictelephone network or a broadcasting network. In addition, though notshown in FIG. 1, server 2 and reproducing and editing devices 1 maycommunicate via a dedicated line, such as a LAN (Local Area Network) orover a wireless communication system. Here, as shown in FIG. 1,reproducing and editing devices 1 may be cellular phones, PDAs (PersonalDigital Assistants), personal computers or the like, and in thefollowing description, reproducing and editing devices 1 are assumed tobe cellular phones 1.

Here, a reproducing and editing device 1 which is a multimedia dataprocessing device may be an information communication apparatus, anaudio and video apparatus, a home electronics apparatus, an amusementapparatus and the like.

In addition, an information communication apparatus may be aninformation apparatus such as personal computer or home server, acommunication apparatus such as a telephone, in particular, a portableinformation communication apparatus such as a cellular phone, a PDA(Personal Digital Assistant), or a laptop type or wearable typecomputer.

In addition, an audio and video apparatus may be a digital still camerawhich is an image recording apparatus, a digital video camera, an HDD(Hard Disk Drive) recorder, a DVD (Digital Versatile Disk) recorder or avideotape recorder which is a video recording apparatus, a DVD player ora video player which is a video reproducing apparatus, a voice recorderwhich is a sound recording apparatus, a silicon audio recorder orplayer, an MD (Mini Disk) recorder or player, a CD (Compact Disk)recorder or player which is a music recording or reproducing apparatus,and a television set or a radio which is a broadcasting data receivingand reproducing apparatus.

In addition, a home electronics apparatus may be an apparatus such as arefrigerator, a washing machine, an air conditioner, a vacuum cleaner,an air purifier, a microwave oven or the like.

In addition, an amusement apparatus may be a game apparatus for home useor a portable game apparatus.

In addition, reproducing and editing device 1 which is a multimedia dataprocessing device may be an apparatus that has more than one functionfrom among the above-described information processing, communication,image recording and reproducing, video recording and reproducing, soundrecording and reproducing, music recording and reproducing, andbroadcasting data receiving and reproducing, or may be in any form, aslong as it is an apparatus that can reproduce or edit multimedia data inthe similar manner as the above-described apparatuses.

In the present embodiment, a case is described in detail, wherereproducing and editing device 1 is specifically a cellular phone 1.

Here, in the present embodiment, media data indicates the multimediadata that follows below, which is any of image (still picture) data,animated image data, sound data, music data, text data and animationdata, streaming data or broadcasting data that is received from theoutside, or data that is obtained by integrating any of the above.

In addition, in the present embodiment, multimedia data indicates datathat is obtained by integrating one or more types of media data, and thedata may be described in a text style that can be read by a person, ormay be in a binary style that cannot be directly be read by a person.

In addition, in the present embodiment, script indicates a descriptionstyle that regulates attributes such as layout information on a screen,start of reproduction along the time axis, time of completion,reproduction speed and sound volume for each type of media data includedin the above-described multimedia data for display, reproduction andediting of the multimedia data. In addition, it is possible to describecontrol information such as the startup of an external application, theacquisition of information from inside and outside the device, and thetransition between links to other multimedia data in the script.

In addition, in the present embodiment, template indicates a form ofmultimedia data which is formed by including the above-described scriptand which can be reproduced and edited, where media data can be added,changed and deleted by rewriting the description of a portion thatrefers to the media data within the script, with user operation or thelike as the trigger. The template may include one or more pieces ofmedia data that is referred from the script.

Next, with reference to FIG. 2, a cellular phone 1 that is included in adata communication system of the present embodiment is formed of aninput/output unit 140 which is a user interface, a control unit 120which is formed of a CPU (Central Processing Unit) or the like and whichcontrols the entirety of this cellular phone 1, a communication unit 110for communicating with another device, and a memory unit 130 which isformed of a ROM (Read Only Memory) or a RAM (Random Access Memory) andwhich stores a program that is executed in control unit 120,intermediate data of such a program, and data that has been receivedfrom another computer.

Furthermore, the above-described input/output unit 140 includes a keycode inputting device 142 that includes numeral buttons such as “1” and“2” and direction buttons such as “R” and “L,” a display 144, such as anLCD (Liquid Crystal Display) for displaying information for a user, amicrophone 146 into which speech sound is inputted, a speaker 148 fromwhich speech sound is outputted, and a camera 149 for taking an imagethat then is inputted.

In addition, control unit 120 includes a timer inside.

In such a cellular phone 1, processing for reproducing and editing atemplate that is obtained by the user from server 2 is carried out bymeans of the hardware of cellular phone 1, the software and the templatewhich are stored in memory unit 130 and which are executed by thecontrol unit 120. Here, the operation of cellular phone 1 itself shownin FIG. 2 is well known.

In addition, server 2 included in the present data communication systemis a server that is formed of a general personal computer or the like,and the detailed description of this configuration is herein omitted.

Next, FIG. 3 shows programs that are stored in memory unit 130 ofcellular phone 1, and the flow of data processing in cellular phone 1.The flow of data processing shown in FIG. 3 is followed in cellularphone 1 when control unit 120 executes each of the programs stored inmemory unit 130. That is to say, control unit 120 executes a programstored in memory unit 130, and thereby, reproducing and editingprocessing is carried out on a template that is received from server 2by communication unit 110.

Concretely speaking, with reference to FIG. 3, first, control unit 120executes a reproducing and editing engine program 131, and thereby, thetemplate that is received in communication unit 110 and that includesthe script and media data is processed. Here, the functionalconfiguration of reproducing and editing engine program 131 is describedbelow.

Next, control unit 120 executes a UI (User Interface) contentsdisplaying program 132 on the basis of the processing results ofreproducing and editing engine 131. UI contents displaying program 132is executed so that the template is processed, and thereby, the contentsof UI in the template are displayed on display 144.

In addition, control unit 120 executes image taking program 133 on thebasis of the processing results of reproducing and editing engineprogram 131, and accepts an input of an image from camera 149 and avariety of instructions from key code inputting device 142. In addition,when a video image is taken, an input of sound from microphone 146 isadditionally accepted.

In addition, control unit 120 executes an image data displaying andreproducing program 134 on the basis of the processing results ofreproducing and editing engine program 131, and reproduces image datathat is included in the template on display 144. In addition, in thecase where sound is included in this image data, this sound isreproduced by speaker 148.

In addition, control unit 120 executes a sound data reproducing program135 o the basis of the processing results of reproducing and editingengine program 131, and reproduces sound data that is included in thetemplate in speaker 148.

In addition, control unit 120 executes a sound data recording program136 on the basis of the processing results of reproducing editing engineprogram 131, and accepts an input of sound from microphone 146 and avariety of instructions from key code inputting device 142.

In addition, control unit 120 executes an animation data displaying andreproducing program 137 on the basis of the processing results ofreproducing and editing engine program 131, and reproduces animationdata that is included in the template on display 144.

In addition, control unit 120 executes a text data displaying andreproducing program 138 on the basis of the processing results ofreproducing and editing engine program 131, and reproduces text datathat is included in the template on display 144.

In addition, control unit 120 executes a text editing program 139 on thebasis of the processing results of reproducing and editing engineprogram 131, and accepts text information and a variety of instructionsfrom key code inputting device 142.

In addition, control unit 120 executes a WWW (World Wide Web) browserprogram 151 on the basis of the processing results of reproducing andediting engine program 131, and reproduces WWW contents.

In addition, control unit 120 executes a WWW browser program 151 on thebasis of the processing results of reproducing and editing engineprogram 131, and acquires data from server 2 via communication unit 110.

In addition, control unit 120 executes an e-mail program 152 on thebasis of the processing results of reproducing and editing engineprogram 131, and accepts text information and a variety of instructionsfrom key code inputting device 142 so as to transmit an e-mail.

In addition, control unit 120 executes a conversation program 153 on thebasis of the processing results of reproducing and editing engineprogram 131, and commences conversation.

In addition, control unit 120 executes a position acquiring program 154on the basis of the processing results of reproducing and editing engineprogram 131, and acquires the present position of the terminal from aGPS (Global Positioning System) 150.

In addition, control unit 120 executes an address book program 155 onthe basis of the processing results of reproducing and editing engineprogram 131, and acquires e-mail addresses and telephone numbers storedin memory unit 130.

In addition, control unit 120 executes a file acquiring program 156 onthe basis of the processing results of reproducing and editing engineprogram 131, and acquires specific media type contents data that hasbeen stored in memory unit 130.

In addition, control unit 120 executes reproducing and editing engineprogram 131 so as to process the template that includes the script andmedia data on the basis of information that has been accepted byexecuting the above-described image taking program 133, sound datarecording program 136, text editing program 139, WWW browser program151, position acquiring program 154, address book program 155 and fileacquiring program 156.

Control unit 120 executes reproducing and editing engine program 131that is stored in memory unit 130, and thereby, the respective functionsshown in FIG. 4 are implemented in cellular phone 1. With reference toFIG. 4, first, a script reading unit 1311 reads the script that isincluded in the template that is received by communication unit 110.Then, script reading unit 1311 passes the script that has been read into a syntax analyzer 1312.

Next, syntax analyzer 1312 analyzes the syntax of the script that hasbeen passed from script reading unit 1311. Then, syntax analyzer 1312passes the results of analysis to an implementation controlling unit1313 and a time managing unit 1314.

Next, time managing unit 1314 prepares a timetable on the basis of theresults of analysis that have been passed from syntax analyzer 1312.Then, time managing unit 1314 passes the prepared timetable toimplementation controlling unit 1313.

In addition, an event managing unit 1315 detects the occurrence of anevent. Then, event managing unit 1315 passes the results of detection toimplementation controlling unit 1313.

Next, implementation controlling unit 1313 carries out the script on thebasis of the timetable that has been passed from time managing unit1314, the results of detection of the occurrence of an event that havepassed from event managing unit 1315, and the results of analysis thathave been passed from syntax analyzer 1312. In addition, implementationcontrolling unit 1313 instructs a script writing unit 1316 to rewritethe script on the basis of the results of detection of the occurrence ofan event that has been passed from event managing unit 1315.

Script writing unit 1316 rewrites the script on the basis of theinstruction from implementation controlling unit 1313 and passes therewritten script to implementation controlling unit 1313. Then, therewritten script is implemented in implementation controlling unit 1313.

Here, a case where a template that has been provided by server 2 isprocessed in cellular phone 1 according to the present embodiment isdescribed, wherein a portion or the entirety of the script that isincluded in the template and media data may be stored in memory unit 130of cellular phone 1, and in this case, the respective units shown inFIG. 3 or 4 processes the script or media data that has been read outand acquired from memory unit 130.

Next, a script which is provided to cellular phone 1 by server 2 andwhich is included in a template that is processed in cellular phone 1 isdescribed. With reference to FIG. 5, the script that is processed incellular phone 1 according to the present embodiment is characterized bybeing formed of a drawing information describing portion and a usernavigation information describing portion.

In the drawing information describing portion, image (still image) data,animated image data, sound data, music data, text data, animation data,streaming data and information (file name, URL (Uniform ResourceLocators), frequency and channel of broadcasting stations) referring tomedia data such as broadcasting data are described.

Here, a describing style such as data URL, prescribed by RFC2397 isutilized, and thereby, the substance of media data itself, as opposed toinformation referring to media data, may be described in the drawinginformation describing portion.

In addition, a script definition inputting event for implementing a linkmovement, startup of an application, display of help and the like,correspondence to an action (operation) and implementation conditionsare described in the user navigation information describing portion.

As for examples of the above-described action, taking video or stillimages by means of startup of a camera, acquisition of user inputtedtext data by means of startup of a text editor, acquisition of sounddata by means of startup of a voice recorder, acquisition of media datafile by means of startup of a file selecting application, automaticacquisition of time information, position information, address book,schedule information, and other information from the inside or theoutside of the apparatus, startup of the application of an apparatus,such as a WWW browser or a mailer, the call of a function inherent to anapparatus such as a telephone call, a melody reproduction, turning on alight or the like, a link transition to the implementation of anotherscript or oscillation of a control signal for controlling anotherapparatus, such as switching of channels of a TV or temperature settingof an air conditioner can be cited.

Furthermore, the embodiment is characterized by including chronologicaldescriptions in both the drawing information describing portion and theuser navigation description portion. That is to say, the embodiment ischaracterized in that information referring to media data ischronologically described in the drawing information describing portion,and events are chronologically described in the user navigationinformation describing portion.

In such a configuration, multimedia data that allows chronologicalreproduction of media data, such as image (still image) data, musicdata, text data, animation data, streaming data and broadcasting dataone after another can be described in a script.

Furthermore, events can be described chronologically, and thereby, forexample, an event such as startup of a camera can be implemented at apredetermined time or in a predetermined time slot while reproducing ascript. Furthermore, such a camera startup event is described so as tocorrespond to an action that is replaced with a predetermined image, andthereby, an application can be implemented in such a manner that when auser presses down a predetermined key in a time slot where theabove-described predetermined image is displayed so that a camera isimmediately started up while pausing the reproduction of the script, andat this point, the operation of taking an image is carried out, theimage that has been taken at this time is displayed in a predeterminedposition when the mode returns to the reproduction of the script.

As described above, an event that starts up along the time sequence andan action that corresponds to this are clearly prescribed within ascript, and thereby, the operation procedure can be simplified,increasing convenience for the user, even in the case where suchmultimedia data as that where a number of pieces of media data arereproduced chronologically by utilizing functions such as camera,microphone and address book in a portable terminal or the like isprepared.

FIGS. 6 and 7 show a concrete example of a script which described below.

With reference to FIG. 6, first, lines 12 to 15 of the header portion(lines 2 to 16) of the script define the contents of the action. That isto say, with reference to FIG. 6, lines 12 to 15 define identificationinformation inherent to each action (action id), implementation style(type) of this action, the action contents (dest) and the implementationregion of the action (Image) as the contents of the action.

Concretely speaking, according to line 12 of FIG. 6, the action of whichthe identification information is act_movie is defined as havingcontents which replace information that is displayed in the “Image”region with the file that is stored in “movie.”

In addition, line 5 of FIG. 6 describes a suggestion as to how thescript can be utilized in cellular phone 1. That is to say, the scriptis defined as a template navigates the contents editing operation. Inaddition to this, the script can be defined as the contents exclusivelyfor reference which do not navigate the editing operation.

Next, with reference to FIG. 7, media data and actions that areimplemented in parallel are described in the paragraph of lines 18 to 39of the body portion (lines 17 to 40) of the script. Furthermore, lines19 to 37 describe that these media data and actions are implementedchronologically. Furthermore, the media data and actions which areimplemented between the 0^(th) second to the 8^(th) second, between the8^(th) second to the 15^(th) second and between the 15^(th) second tothe 19^(th) second are described in the paragraphs of lines 20 to 25,lines 26 to 31 and lines 32 to 36, respectively.

Concretely speaking, lines 21 to 23 of FIG. 7 describe that image data,text data and sound data are reproduced between the 0^(th) second andthe 8^(th) second, and line 24 describes that the action of which theidentification information is act_movie is implemented in the case wherethe UI of “movie startup” is displayed and the select key has beenoperated. Line 30 describes that the UI of “sound recording” isdisplayed, and then, two seconds later, a notification is received fromthe timer and the action of which the identification information isact_speech is implemented. Here, the name that is described following“src=” in lines 28 to 30 is the source ID that represents the origin ofthe occurrence of the event. The display of the UI may be described inthe script, as shown in FIG. 7, or may be displayed by automaticallyreading out a table that prescribes the correspondence between thestartup application and the UI that is displayed at this time, as shownin FIG. 8, when this table has been stored in advance in memory unit 130of cellular phone 1, and control unit 120 of cellular phone 1 executesreproducing and editing engine program 131.

As described above, the script that is processed in cellular phone 1according to the present embodiment is characterized by being formed ofa drawing information describing portion and a user navigationinformation describing portion which describe drawing information anduser navigation information, which are chronologically carried out.

Here, though in the present embodiment, a case is described where ascript in a text style that can be read by a person is processed incellular phone 1, the script may be in a binary style that cannotdirectly be read by a person.

In addition, though in the present embodiment, a case is described wherea unified script is processed in cellular phone 1, a script may beformed so as to be divided into a drawing information describing portionthat includes information referring to media data and a user navigationinformation describing portion, and in such a configuration, only aportion of a script where the drawing information describing portionthat includes information referring to media data is described is editedat the time when media data is replaced or inserted.

Concretely speaking, FIGS. 70 and 71 show scripts, each of which isformed so as to be divided into a drawing information describing portionand a user navigation describing portion, which are scripts wheredrawing information describing portions are described and scripts whereuser navigation information describing portions are described.

Concretely speaking, the script where the drawing information describingportion shown in FIG. 70 is described describes that still image top.jpgand text top.txt are displayed for 0 to 2 seconds after the start ofimplementation of the script, and still image image.jpg, music databgm.amr and text message.txt are reproduced for 2 to 8 seconds after thestart of implementation of the script.

In addition, the script where the user navigation information describingportion shown in FIG. 71 is described is a script that carries out thefunction of replacing and inserting image.jpg and message.txt from amongmedia data of which the reference information is described in FIG. 70 asthe information to be referred to.

FIGS. 72A to 72D show concrete examples of screen displays, where theabove-described script shown in FIGS. 70 and 71 is implemented aftersyntax analysis in cellular phone 1 according to the present embodiment.First, as shown in FIG. 72A, an image file selecting form, together witha letter sequence of “SELECT PHOTO,” as well as a letter sequenceinputting form together with a letter sequence of “ENTER BIRTHDAYMESSAGE,” are displayed in accordance with the description of FIG. 71. Auser selects an image file in the screen shown in FIG. 72A, and thecondition when the text is inputted is shown in FIG. 72B. When the“send” button is selected and operated in the screen shown in FIG. 72B,the image file that has been selected at this time is incorporated intoa template as “image.jpg” and the inputted text is incorporated into thetemplate as “message.txt,” so as to be reproduced and implemented. Thatis to say, still image top.jpg and text top.txt are displayed as shownin FIG. 72C for 0 to 2 seconds after the start of the reproduction ofthis script, and subsequently, image.jpg and message.txt are displayed,as shown in FIG. 72D, for 2 to 8 seconds after the start of thereproduction of the script, and at the same time, music data bgm.amr isreproduced. Image.jpg displayed in FIG. 72D is an image selected by theuser on the screen shown in FIG. 72B, and message.txt is a text that isinputted on the screen shown in FIG. 72B.

As described above, a script is formed so as to be divided into adrawing information describing portion and a user navigation describingportion, and thereby, a merit can be obtained, where in the case asdescribed below where the user navigation describing portion is deletedfrom the script and transmitted by being attached to an e-mail, only thedrawing information describing portion may be attached, and the processfor analyzing the script taking which portion is to be deleted intoconsideration can be omitted.

Here, in order to relate the script that includes a drawing informationdescribing portion to the script that includes a user navigationdescribing portion, it is preferable to separately prepare a script thatincludes a describing portion which connects the two scripts to eachother as a script such as, for example, that shown in FIG. 73, and it ispreferable to handle a set of these scripts and a media data group thatis referred from the script that includes the drawing informationdescribing portion as a template.

In addition, a link to the script that includes the corresponding usernavigation describing portion is described within the script thatincludes the drawing information describing portion, and thereby, theoperation can be shifted to the implementation of the script thatincludes the user navigation describing portion according to apredetermined timing while reproducing the script that includes thedrawing information describing portion, so that the screen can beswitched to the inputting screen, as shown in FIG. 72A.

Next, reproducing and editing processing of a script in cellular phone 1according to the present embodiment is described with reference to theflowchart of FIG. 9. Control unit 120 of cellular phone 1 reads outreproducing and editing engine program 131 of which the configuration isshown in FIG. 4 from memory unit 130 and implements this program, andthereby, the process shown in the flowchart of FIG. 9 is implemented.

With reference to FIG. 9, first, initialization is implemented in stepS11, and after that, a script is acquired from server 2 via the internetby communication unit 110 in step S12.

Next, in step S13, control unit 120 executes reproducing and editingengine program 131 and reads in the script that has been acquired byscript reading unit 1311.

Next, in step S14, syntax analyzer 1312 carries out syntax analysis onthe read in script. Here, the method for syntax analysis is not limitedto one specifically for the present invention, but rather, may be ageneral syntax analyzing method.

Next, in step S15, time managing unit 1314 prepares timetables, as shownin FIGS. 10 to 12, on the basis of the results of syntax analysis. FIG.10 is a diagram showing a concrete example of a timetable of a mediadrawing process that has been prepared on the basis of the drawinginformation describing portion of the script shown in FIGS. 6 and 7, andFIGS. 11 and 12 show concrete examples of an event processing timetableand a UI allocation state displaying timetable, which have been preparedon the basis of the user navigation information describing portion ofthe script shown in FIGS. 6 and 7. That is to say, in step S15, timemanaging unit 1314 prepares correspondence between script implementationtime and media data that is reproduced at this time, correspondencebetween script implementation time and a process in accordance with theevent that has occurred at this time, and correspondence between scriptimplementation time and allocation of UI's which are operated at thistime in timetable style, as shown in FIGS. 10 to 12, on the basis of thescript that has been analyzed in step S14.

Concretely speaking, the timetable of a media drawing process shown inFIG. 10 is a timetable showing that animated image 1 (FirstImage.mpg),text 1 (FirstText.txt) and music 1 (FirstSound.smf) are reproduced from0 to 8 seconds after the start of the implementation of the script, text2 (SecondText.txt) is reproduced from 8 to 15 seconds after the start ofthe implementation of the script, and still image 3 (Thirdlmage.jpg),text 3 (ThirdText.txt) and music 3 (ThirdSound.smf) are reproduced from15 to 19 seconds after the start of the implementation of the script onthe basis of the drawing information describing portion of the scriptshown in FIGS. 6 and 7.

Here, in the case where a text that is included in a script and a textthat is inputted by the user is displayed so as to be overlapped withother images or videos in the same region, it is preferable toautomatically adjust attributes such as color, font and the like, inorder to enhance the visibility of the text. The simplest implementationof such an automatic adjustment is obtained by using the complementarycolor (color that has the complement of one of each component value ofRGB of the background as an RGB component) of the color attributes ofthe background as the color attributes of the text, in the case wherethe distance √((ΔR)²+(ΔG)²+(ΔB)²) of the color attributes between thetext and the background is smaller than the predetermined thresholdvalue when concrete expressions of the color attributes are made by acombination of three values of RGB.

In addition, the event processing timetable shown in FIG. 11 is atimetable that shows that an animated image taking application that isimage taking program 133 starts up when the select key is operatedbetween 0 and 8 seconds after the start of the implementation of thescript, a screen for selecting one of an animated image takingapplication that is, for example, image taking program 133 and a letterinputting application that is text editing program 139 is displayed ondisplay 144, and a program that is selected from among these starts upwhen the select key is operated between 8 and 15 seconds after the startof the implementation of the script, and a sound inputting applicationthat is sound data recording program 136 starts up by means of a timerbetween 8 and 15 seconds after the start of the implementation of thescript, on the basis of the user navigation information describingportion of the script shown in FIGS. 6 and 7.

In addition, the UI allocation state displaying timetable shown in FIG.12 is a timetable showing that display 144 displays “take movie” as thedescription of the select key, in order to show the user that theanimated image taking application that is image program 133 starts upwhen the select key is operated between 0 and 8 seconds after the startof the implementation of the script, display 144 displays “take imagewith camera,” “input text” as the description of the select key, inorder to show the user that either the animated image taking applicationthat is image taking program 133 or a letter editing application that istext editing program 139 starts up when the select key is operatedbetween 8 and 15 seconds after the start of the implementation of thescript, and display 144 displays “recording” as the description ofstartup by means of the timer, in order to show the user that a soundinputting application that is sound data recording program 136 starts upby means of the timer between 8 and 15 seconds after the start of theimplementation of the script.

Next, in step S16, implementation controlling unit 1313 implements thescript on the basis of the results of the syntax analysis and thetimetable. In addition, at the time of the implementation of the script,implementation controlling unit 1313 instructs UI contents displayingprogram 132 to display the UI that varies along the time ofimplementation of the script on display 144 on the basis of the UIallocation state displaying timetable. Here, the process forimplementing the script is described below, by citing concrete examplesusing as a base the script that is shown in FIGS. 6 and 7.

Then, when, in step S17, event managing unit 1315 detects the occurrenceof an event (YES in S17), in step S18, a process that corresponds tothis event is carried out in implementation controlling unit 1313. Here,the process that corresponds to the event is described below, by citinga subroutine.

The events that are detected in step S17 are primarily divided intothree types, system events that are terminal reaching events where theimplementation of the script reaches the terminal of the implementationtime at the time when the script is implemented chronologically, stableinputting events for implementing a stable process, such as indicationof a menu display, and script definition inputting events that indicatelink shift, application startup, display of help or the like. Concretelyspeaking, inputs of indications by the user or the like from key codeinputting device 142 and the like during the implementation of thescript correspond to stable inputting events and script definitioninputting events. Therefore, the implementation of the script iscontinued until certain events, such as inputs of an indication from keycode inputting device 142, occur during the implementation of thescript.

At this point, reproducing and editing processing of the script incellular phone 1 is completed.

Here, in the case where the above-described reproducing and editingprocessing is interrupted by a conversation program or the like duringprocessing, or is discontinued for the convenience of the user or as aresult of depletion of the battery, the script during reproducing andediting and the script during modification are stored in memory unit 130so that processing can be discontinued, and it is preferable for theconfiguration to, at the time when it is later resumed, allow the scriptduring reproduction and editing and the script during modification to beread out from memory unit 130 so as to be resumed from the point duringreproduction and editing where it was stopped.

Furthermore, reproducing and editing processing in the case where thescript is written in SMIL (Synchronized Multimedia Integration Language)is concretely described using the flowchart of FIG. 79.

With reference to FIG. 79, first, the script that is written in SMILincluded in the template is analyzed (parsed) so as to be stored in thedata structure (S2401).

FIGS. 80A and 80B show a concrete example of a data structure where theresults of analysis are stored in step S2401. As shown in FIG. 80B, instep S2401, an instance in the DisplayContent class for holdingattribute values, such as types of media, file names and displayregions, is prepared for each piece of media data of which the referenceinformation is described in the script and referred to, and theinstances are unified into a set, which is then managed inDisplayScheduler shown in FIG. 80A.

Furthermore, the results of analysis of the script in step S2401 areshown in FIG. 81. FIG. 81 concretely shows the results of the analysisof the script shown in FIGS. 6 and 7, and 11 instances in theDisplayContent class are prepared as the results of analysis of thescript. This number indicates the number of pieces of media data whichare referred to from reference information that is finally described inthis script.

Here, the files of the media data (Mediafile) designated by instances 5,6 and 7 from among these 11 instances are empty, and show that theinitial condition shown in FIGS. 6 and 7 is a condition where thecorresponding media data is not referred to. The reference informationis described by accepting the designation of the media data in the belowdescribed editing process, and the media data is referred to.Alternatively, a guidance can be display by designating a predeterminedfile. In the case where instance 5 is cited as an example, the actioncontents have been found by analysis to be image taking and insertion bymeans of the startup of a camera, and thus, the illustration image ofthe camera is displayed as guidance by designating the illustrationimage of the camera (camera.jpg). In addition, in a simpler manner, aletter sequence which is the value of EvDesc (such as “startup ofcamera”) may be displayed in the bottom line of the screen as a guidancedisplay.

Then, this script is reproduced (S2402) while following the datastructure as shown in FIGS. 80A and 80B.

In the case where a process for insertion and switching of media data iscarried out as described below (YES in S2403), the corresponding valueis updated (S2404) in the data structure that stores the results ofanalysis in step S2401. Here, rewriting processes, such as processes forinsertion and switching of media data are described in detail below.

FIG. 82 shows a concrete example of the results of analysis of thescript that has been updated in step S2404. The results of editing shownin FIG. 82 are the results where rewriting processing has been carriedout on the script shown in FIGS. 6 and 7 in the initial condition shownin FIG. 81. Concretely speaking, a case is shown where theabove-described rewriting processing is carried out and media data“CameraPict01.jpg” “EditText01.txt” and “Voice01.amr” are added to thescript in the initial state, and these pieces of reference informationare described in corresponding instances 5, 6 and 7. Here, the file nameis prescribed on the script side on the basis of image data“FirstImage.mpg,” and therefore, the movie file that is obtained as aresult of the taking of a movie is also stored under this name. That isto say, the substance of the file is rewritten and updated without thefile name being rewritten.

The processes of steps S2402 to S2404 are repeatedly carried out (NO inS2405) until the implementation of this script has reached thetermination, and when the implementation reaches the termination (YES inS2405), the reproducing processing of the script is completed, andwhether or not the script is transmitted by being attached to an e-mailis determined (S2406).

In the case where insertion of media data or editing of a script, suchas inserted and switching process of media data, are not completed (NOin S2406), the process is returned to step S2402, and the implementationof this script is reproduced from the beginning.

Meanwhile, in the case where editing is completed and the script istransmitted by being attached to an e-mail (YES in S2406), the scriptthat is written in SMIL is generated from the data structure shown inFIG. 82 (S2407). Then, the script that is written in SMIL and has beengenerated in step S2407 is transmitted by being attached to an e-mail,or is stored (S2408). Here, the process for attachment to an e-mail instep S2408 is also described below.

FIG. 83 shows a concrete example of a script that is transmitted in stepS2408. The script shown in FIG. 83 is a script on which rewritingprocessing is carried out so that media data such as image data, textdata and sound data are added to the script in the initial state shownin FIGS. 6 and 7, and, concretely speaking, the underlined portions inFIG. 83 are reference information of media data that has been insertedby means of a rewriting process.

In addition, at the time when a script is transmitted, the editingfunction that is a user navigation information describing portion may bedeleted and transmitted. The script shown in FIG. 84 is a script that isobtained by carrying out rewriting processing for adding media data suchas image data, text data and sound data to the script in the initialstate shown in FIGS. 6 and 7, and by subsequently deleting the editingfunction, and, concretely speaking, the definition portion of the actioncontents described in <action id= . . . > and the portion thatcorresponds to the event and the action to each other described in<event ref= . . . > has been deleted. This is shown in FIG. 84 byunderlining.

Here, when the script written in SMIL is read out from the datastructure shown in FIG. 82 in step S2407 so as to generate the scriptshown in FIG. 84, the attributes concerning Event and Action, which areincluded in the data structure, are ignored in the case where theediting function is deleted. Alternatively, though in the case where thescript that includes the editing function is written out, “insert” hasbeen designated as the action contents (Act Type=“insert”), thereference information of media data has already been described as aresult of the above-described process, and therefore, the actioncontents are changed to “replace” (Act Type=“replace”). This is shown inFIG. 83 by underlining.

Next, a process that corresponds to the event that is carried out instep S18 is described using the flowchart of FIG. 13.

With reference to FIG. 13, implementation controlling unit 1312 pausesthe script that is implemented in step S21 when detection information ofthe occurrence of the event is received from event managing unit 1315 instep S117.

Then, in the case where the event that has been detected in step S17 isan event that indicates that the implementation of the script hasreached the termination of the implementation (YES in S22),implementation controlling unit 1313 completes the implementation ofthis script in step S23, and the process returns to step S16 of FIG. 9,so that the implementation of the script is started again. That is tosay, the implementation of the script is returned to the beginning ofthe implementation again, and the implementation of the script isrepeated from the start. Here, though in the present embodiment, thescript that is included in the acquired template is repeatedlyimplemented, the implementation of the script may be completed afterthis has been implemented one or more predetermined times.Alternatively, choices as to whether the implementation of the script isto be repeated or completed are proposed on the screen so that therepetition or the completion of the implementation of the script may beinstructed through input by the user by means of, for example, buttonoperation.

In the case where the event that has been detected in step S17 is anevent other than the event that shows the implementation of the scripthas reached the terminal of the implementation (NO in S22),implementation controlling unit 1313 determines process contents in stepS24 on the basis of the event that has been detected, and carries outthe determined process in steps S25 to S27. Here, the process fordetermining the process contents in step S24 and the respectiveprocesses shown in steps S25 to S27 are described below by citing therespective flowcharts.

At this point the process that corresponds to the event in step S18 iscompleted, and the process is returned to the reproducing and editingprocess shown in the flowchart of FIG. 9.

Next, the process of determining the process contents in step S24 isdescribed with reference to the flowchart of FIG. 14.

With reference to FIG. 14, when the occurrence of an event is detectedin step S17, implementation controlling unit 1313 refers to the timeparameter value (S31) at this time, that is to say, the parameter thatrepresents the time of implementation of this script, the source ID(S32), that is to say, the identification information of the origin fromwhich the event occurs, and the event occurrence processing timetable(S33) that is prepared by time managing unit 1314, and thereby,determines the event that has occurred. Furthermore, the processingcontents that are made to correspond to this event at the time of theoccurrence of the event are determined from the determined event and theevent occurrence processing timetable. Here, in the case where two ormore processes are determined (YES in S34), implementation controllingunit 1313 displays a screen for selection on display 144 or the like instep S35, and accepts the selection of the process that is started fromthe present time from among the two or more processes. In the casewhere, for example, “invoke camera” and “local folder” are proposed tothe list that is displayed on the screen for selection, as shown in FIG.54A, and “invoke camera” is selected, the camera is started up, and inthe case where “local folder” is selected, as shown in FIG. 54B, thedata folder is accessed.

Alternatively, a configuration may be provided where, in step S35,implementation controlling unit 1313 may implement all of the determinedtwo or more processes sequentially, one by one. In such a case, theprocesses are implemented in the order in which the scheduledescriptions of the events that correspond to these processes appear.Concretely speaking, when time between the 8^(th) and 15^(th) second inthe script of FIG. 7 is cited as an example and in the case where theselect key is pressed, “startup of camera” which is described in line 28in accordance with the order of descriptions is first implemented, andafter that, “text input” that is described in line 29 is implemented.Here, the priority order of the implementation of the two or moreprocesses is described in the script so that the two or more processesmay be implemented sequentially following this order in theconfiguration.

Alternatively, in step S35, in the case where there is a shortcut meansfor respectively implementing the two or more processes, the processthat corresponds to the shortcut means designated by the user may bedirectly implemented in the configuration. Concretely speaking, when thetime between the 8^(th) and 15^(th) second in the script of FIG. 7 iscited as an example, in the case where the key for starting up thecamera is pressed down, “camera startup” that is described in line 28 isimplemented, while in the case where the key for text input is presseddown, “text input” that is described in line 29 is implemented, in theconfiguration.

At this point, the process that corresponds to the event in step S24 iscompleted, and the process returns to the event processing that is shownin the flowchart of FIG. 13.

Furthermore, reference to the event timetable in step S33 is describedusing the flowchart of FIG. 27.

With reference to FIG. 27, in the case where the referred source ID is“Timer” (“source ID”=TIMER in S131), the corresponding processes aredetermined with reference to the column of “Timer” in the eventprocessing time table and the time parameter value (S132).

Meanwhile, in the case where the referred source ID is “UI” (“sourceID”=UI in S131), the column of the UI in the event processing timetable,type ID, that is to say, the identification information inherent to thekey that is operated at the time of the occurrence of the event and thetime parameter value are referred to (S133), and the correspondingprocesses are determined (S134).

Here, the identification information of the hardware key of the terminalmay be directly utilized as the identification information inherent tothe above-described key that is described in the event processingtimetable and the script, or the logic symbol may be utilized in amanner where, as shown in FIG. 28, the table that prescribes thecorrespondence between the logic symbol and the hardware key is storedin advance in memory unit 130 of cellular phone 1, and implementationcontrolling portion 1313 may read out this table at the time when theevent timetable is referred to so that the logic symbol is referred tofrom the operated hardware key, and thereby, the event may bedetermined.

At this point, reference to the event timetable in step S33 iscompleted, and the process returns to the process contents determiningprocess shown in the flowchart of FIG. 14.

In the case where the event of which the occurrence is detected in stepS17 is determined to be an event that implements the menu processing inthe above-described process, the process shown in FIG. 15 is implementedin step S25. That is to say, with reference to FIG. 15, first,implementation controlling unit 1313 displays a menu screen as shown inFIG. 16 on display 144 in step S41, and accepts the selection of theuser. Concretely speaking, with reference to FIG. 16, selections fromamong the preview processing of this script, storage processing in adata folder, a process for storing media data that is referred from thisscript in an individual media file, a process for attaching this scriptand media data to an e-mail, a process for transmitting this script andmedia data to an MMS (Multimedia Messaging Service), a process forediting the slide during reproduction of this script, a process fordisplaying the properties of this script, a process for completing theapplication and the like are accepted. Then, in step S42, the processthat corresponds to the selection of the user is implemented. Here, theslide represents a portion of the script that is implemented in parallelas described in lines 20 to 25 in FIG. 7. That is to say, the scriptthat is shown in FIG. 7 is formed of three slides of lines 20 to 25,lines 26 to 31 and lines 32 to 36, which are reproduced sequentially.

Furthermore, in the case where “slide editing” is selected in a menuscreen, as shown in FIG. 16, implementation controlling unit 1313displays a menu screen for editing slides, as shown in FIG. 36, ondisplay 144 in step S41, and accepts a selection of a slide deletingprocess or a slide rewriting prohibiting process.

Here, whether slide editing is permitted or prohibited is described inthe script, and in the case where it is prohibited, slide editing maynot be carried out without proposing choices of the above-describedslide editing in the configuration.

Concretely speaking, an example of a script where whether such slideediting is permitted or prohibited is described is shown in FIG. 68, andthe process of determining a permission that implements editing ofslides in step S42 is described using the flowchart of FIG. 69. That isto say, with reference to FIG. 69, in the case where meta-description(name=“readOnly”), which is edition prohibition, as shown in line 4 ofFIG. 68 is described in the script (YES in S2201), the attributes(contents) that describe the prohibition are referred to. Then, in thecase where the attributes are “yes” (YES in S2202), editing of the slideis prohibited in step S42. Meanwhile, in the case where the attributesare not “yes” (NO in S2202), editing of the slide in step S42 ispermitted. In addition, even in the case where there is nometa-description of editing prohibition from the start in this script(NO in S2201), editing of the slide is permitted in step S42.

At this point, the descriptions of the slide editing permissiondetermination are completed, on the basis of concrete examples.

In addition, the process in step S42 in the case where slide deleting isselected on a menu screen for slide editing as shown in FIG. 36 in stepS41 is described using the flowchart of FIG. 35.

With reference to FIG. 35, when a process for deleting the slide duringreproduction of this script is selected in step S41, implementationcontrolling unit 1313 refers to the time parameter value at this time, amedia drawing processing timetable and the event processing timetable(S171 to S173), and determines the media drawing processing and theevent processing that are effective at this point in time. Then, theschedule describing portion of the media drawing processing and theevent processing that have been determined to be effective at this pointin time is deleted from the script, and thus, this script is rewritten(S174).

At this point, the process in step S42 in the case where the process ofdeleting the slide is implemented is completed, and the process returnsto the menu process shown in the flowchart of FIG. 15.

In addition, the process in step S42 in the case where slide rewritingprohibition is selected on a menu screen for slide editing as shown inFIG. 36 in step S41 is described using the flowchart of FIG. 37.

With reference to FIG. 37, when the rewriting prohibition process in theslide during reproduction of this script is selected in step S41,implementation controlling unit 1313 refers to the time parameter valueat this time and the event processing timetable (S181 and S182), and theevent processing that is effective at this point in time is determined.Then, the schedule describing portion of the event processing that hasbeen determined to be effective at this point in time is deleted fromthe script, and then, this script is rewritten (S183).

At this point, the process in step S42 in the case where rewriting ofthe slide is prohibited is completed, and the process returns to themenu process shown in the flowchart of FIG. 15.

In addition, in the case where the event of which the occurrence hasbeen detected in step S17 is determined to be an event that implements alink shifting process, in the above-described process, the process shownin FIG. 17 is implemented in step S27. That is to say, with reference toFIG. 17, first, in step S51, implementation controlling unit 1313 refersto the parameter that indicates the script that is linked which isincluded in the script, and interrupts the implementation of this scriptin step S52. Then, in step S53, the script that is linked is acquired onthe basis of the parameter that is referred to in step S51. After that,the process is returned to step S13, where the process shown in FIG. 9is implemented, and the acquired script is read out so as to beimplemented in the same manner.

In addition, in the case where the event of which the occurrence isdetected in step S17 is determined to be an event that implements anapplication implementing process in the above-described process, theprocess shown in FIG. 18 is implemented in step S26. That is to say,with reference to FIG. 18, first, implementation controlling unit 1313displays a screen for selecting whether or not the application is to bestarted up on display 144. In the case where startup of the applicationis selected (YES in S61) implementation controlling unit 1313 starts upa predetermined application in step S62. Next, in the case wherepredetermined data needs to be acquired from this application (YES inS63), this data is acquired in step S64. Then, in the case where thisscript which is being implemented needs to be rewritten (YES in S65),script writing unit 1316 implements a writing process of the script instep S66.

Furthermore, implementation controlling unit 1313 resumes thereproduction of the script after the completion of the above-describedprocess of script rewriting. The reproduction of the script can bestarted at the point in time when the reproduction of the script isstarted, at the point in time when the script is reproduced at thispoint in time, or at the point in time when the rewritten section isstarted. Any of the above-described configurations may be provided, orthe configuration may allow a user to select from among theabove-described choices.

Here, the above-described point in time of resumption may be determineddynamically in accordance with a predetermined rule that has been set inthis terminal. In the same manner, time of reproduction of the script orother times concerning the reproduction, such as the point in time ofcompletion, may be determined. In the following, a process fordetermining the point in time of resumption of the script reproductionin accordance with the rule “the script is resumed from the position ofreproduction at the present time in the case where the remainingreproduction time of the script is 10 seconds or less, and the script isresumed from the point in time of the start of the script in othercases” is concretely described using the flowchart of FIG. 67.

With reference to FIG. 67, first, implementation controlling unit 1313refers to the present position of reproduction of the script (S2101).This point of reproduction indicates the point of reproduction at thepoint in time when the script is paused in step S21.

Next, implementation controlling unit 1313 refers to the total time ofreproduction of the script (S2102). The total time of reproduction ofthe script is acquired from the schedule describing portion of thescript. In the case where the script is the one shown in FIG. 7, forexample, in step S2102, the sum of the reproduction time of lines 20 to25, lines 26 to 31 and lines 32 to 36, which are chronologicallyimplemented, is calculated in a manner where the total reproduction timeof the script is acquired as 19 seconds.

Next, implementation controlling unit 1313 subtracts the present pointof reproduction of the script from the total reproduction time of thescript that has been acquired in step S2102, and thereby, the remainingreproduction time of the script is calculated (S2103).

Then, in the case where the remaining reproduction time of the scriptthat has been calculated in step S2103 is 10 seconds or less (NO inS2104), implementation controlling unit 1313 sets the point in time ofthe start of the reproduction as the present point in time of thereproduction of the script. In other cases (YES in S2104) implementationcontrolling unit 1313 sets the point in time of the reproduction startas the point in time of the start of the script.

At this point, the descriptions of the process for determining the pointin time of the reproduction of the concrete script reproduction arecompleted.

Next, the startup of the application in step S62 is described using theflowchart of FIG. 24.

With reference to FIG. 24, in the case where implementation controllingunit 1313 is started up (S102) using the reproduction time and the sizeof the region that have been described in the script as startupparameters in the case where a predetermined application is registeredin memory unit 130 of cellular phone 1 (YES in S101). In the case wherea predetermined application is not registered in memory unit 130 ofcellular phone 1 (NO in S101), and in addition, in the case where dataneeds to be acquired from the application in this script (YES in S103),a local file acquiring application that is a file acquiring program 156is started up (S104).

Here, in the case where a predetermined application is not registered inmemory unit 130 of cellular phone 1 in step S101, a predeterminedapplication is downloaded from server 2 via communication unit 110, andis stored in memory unit 130 of cellular phone 1, and then, theconfiguration may allow startup of the application.

Furthermore, the operation in step S102 is described in the following,using the flowchart of FIG. 66, by citing a case where the applicationthat is started up is an image taking application as a concrete example.

With reference to FIG. 66, first, implementation controlling unit 1313acquires the size of the drawing region on which replacing media data isdisplayed from the script (S2001). The size of the drawing region isgenerally described in the header portion of the script. Concretelyspeaking, when the script is assumed to be the one shown in FIG. 6,numeral values 176 and 120, which are designated as the width and theheight of the drawing region, are acquired from the description in line8 of FIG. 6.

Next, implementation controlling unit 1313 determines the minimum imagesize that can include this drawing region by means of this image takingapplication (S2002) on the basis of the size of the drawing region thathas been acquired in step S2001. Concretely speaking, in the case wherethe image taking application can select width 600×height 480, width120×height 160 or width 120×height 128 as the image size, it isdetermined in step S2002 that the image size width 600×height 480 caninclude the size of drawing region of width 176×height 120, in theconcrete example.

Then, implementation controlling unit 1313 designates the image size inthe image taking application at width 600×height 480 on the basis of thedetermination results in step S2002, and then, starts up the imagetaking application (S2003).

Here, in the case where the application that has started up is a soundrecording application, recording can be carried out for the time that isdescribed in the script, by carrying out the same process as theabove-described process. In addition, in the case where the applicationthat is started up is a file acquiring application, a file of which thesize is greater than the remaining size can be utilized can beprohibited from being selected, so that the sum of the file sizes of themessages does not exceed the upper limit of the size that can betransmitted. In addition, in the case where the application that hasstarted up is a file acquiring application that acquires animatedimages, a file of which the reproduction time is longer than thereproduction time that is described in the script can be prohibited frombeing selected.

At this point, the startup of the application in step S62 is completed,and the process returns to the process for implementing the applicationshown in the flowchart of FIG. 18.

Next, the data acquisition in step S64 is described using the flowchartof FIG. 25.

With reference to FIG. 25, first, implementation controlling unit 1313receives data from the application that is started up in step S62(S111). Then, in the case where this acquisition data is not suitablefor the reproduction time that is described in the script, and the sizeof the region (NO in S112), implementation controlling unit 1313converts the acquisition data to one that is suitable for thedescription of the script (S113).

Here, in step S113, the reference information that has been described inadvance in the script may be used to synthesize the media data that isreferred to and the acquisition data, so as to generate a new script ormedia data. In the case where the acquisition data is media data havingthe reproduction time, this generation process can be described asfollows, using FIGS. 56A to 56C.

As shown in the upper portion of FIG. 56A and in FIG. 56B, in thisconcrete example, “original media data” has the reproduction time of 5seconds, and is initially described so as to be reproduced from time t1.In addition, “acquisition data” is a video that has been taken for 5seconds starting from time t2 (=t1+3), which is a time parameter valueof the event that is referred to in step S31. In this case, when theuser instructs the portion of “original media data” of which thereproduction time corresponds to the time when the acquisition data isbeing taken to be replaced with “acquisition data” by means of anappropriate user interface, the contents of “data that is reproducedafter rewriting” and the time of reproduction become the data that isshown in the lower portion of FIG. 56A and in FIG. 56C. According to onemethod for implementing a process that obtains such effects, apredetermined portion of the original media data where the time oftaking the acquisition data and the reproduction time overlap is storedby rewriting by means of image data taking and editing program 133following the instruction of implementation controlling unit 1313.According to another implementation method, script writing unit 1316that has received an instruction from implementation controlling unit1313 generates and respectively writes out a script that expresses thetime of a new start and the time of reproduction of the original mediadata and acquisition data. According to the latter implementationmethod, the script that is written out by script writing unit 1316 inthe above-described concrete example becomes the script shown in FIGS.56B and 56C. Here, it is preferable to indicate by means of anappropriate user interface whether or not the portion of the acquisitiondata between time t3 and time t4 which does not overlap with theoriginal media data is added to the generated data.

Here, in step S113, new media data may be generated by, for example,taking out a portion of the acquisition data where a process for takingout only sound data from video data is carried out.

In addition, in step S112 and step S113, where the above-describedprocess is carried out, in the case where the size of the memory regionoccupied by the acquisition data exceeds a predetermined value, aprocess for reducing the size of the data, such as reduction of thecolors of the image data or reduction of the sampling rate of the sounddata, may be carried out.

In addition, in step S112, where the above-described process is beingcarried out, in the case where the acquisition data is not suitable forthe description of the script, predetermined media data (default data)that is stored in memory unit 130 of cellular phone 1 may be utilized inplace of the acquisition data.

At this point, the data acquisition in step S64 is completed, and theprocess returns to the process of application implementation shown inthe flowchart of FIG. 18.

Next, the process of rewriting the script in step S66 is described usingthe flowchart of FIG. 26.

With reference to FIG. 26, implementation controlling unit 1313implements the process of rewriting the script in script writing unit1316 (S122) in the case where the sum of the media data and theacquisition data which are included in the script is no greater than thelimit value that is stored in memory unit 130 (YES in S121). In othercases (NO in S121), implementation controlling unit 1313 displays amessage to the effect that script cannot be rewritten on display 144(S123).

Here, in step S122, implantation controlling unit 1313 can modify thepoint in time where the acquisition data is written by utilizing thetime parameter value of the event that is referred to in step S31. Inthe case where a process for rewriting media data by starting up acamera application is carried out as a process for rewriting a script bycarrying out a key operation 4 seconds after the start of thereproduction as shown in FIG. 57B in a script as shown in FIG. 57A,image taking is started at the time when the key operation is carriedout (fourth second), and thereby, video 2 that has been acquired fromthe camera application after the completion of image taking is writteninto the section between the time when the key operation is carried out(fourth second) and the time when the reproduction of video 1 which isthe original media data is completed (eight second). Here, though inthis example, media data is written in at the point in time of the timeparameter value of the event, the media data can be written in at thepoint in time before or after the time parameter value of the event.

At this point, the process of rewriting the script in step S66 iscompleted, and the process is returned to the process of applicationimplementation shown in the flowchart of FIG. 18.

At this point, reproducing and editing processing of the script iscompleted in cellular phone 1 in the present data communication system.

Concretely speaking, a case where reproducing and editing processing ofthe script shown in FIGS. 6 and 7 is carried out in cellular phone 1 inthe present data communication system is described using the flowchartof FIGS. 19 and 20.

With reference to FIG. 19, when line 20 of the script shown in FIG. 7 isimplemented, first, script analysis is carried out in syntax analyzer1312 of reproducing and editing engine program 131, and in step S71,implementation controlling unit 1313 instructs image data displaying andreproducing program 134 to display and reproduce animated image data“FirstImage.mpg” in the “Image” region between 0 and 8 seconds. Then, instep S72, image data displaying and reproducing program 134 displays andreproduces animated image data “FirstImage.mpg” on display 144 between 0and 8 seconds, and furthermore, the sound data is reproduced fromspeaker 148 in the case where the animated image data includes sounddata.

Next, in the case where an event where a user operates the select keyoccurs between 0 and 8 seconds, the process shown in FIG. 20, which is aprocess described on line 24 of the script shown in FIG. 7, isimplemented. That is to say, with reference to FIG. 20, when eventmanaging unit 1315 detects the occurrence of the event where the selectkey is operated, in step S81, implementation controlling unit 1313refers to the event processing timetable, of which a concrete example isshown in FIG. 11, and determines the corresponding processing contents(action). That is to say, in step S82, implementation controlling unit1313 refers to the event processing timetable and starts up image takingprogram 133 and sound data recording program 136, which are thecorresponding applications.

Next, in step S83, image taking program 133 displays the UI that isrequired for image taking on display 144, and generates image data byutilizing camera 149 and key code inputting device 142. In addition, atthe same time as this, sound data recording program 136 utilizesmicrophone 146 so as to generate sound data. Then, a data synthesizingprogram, not shown, synthesizes the generated image data and sound data.As a result of this, in step S83, new data is acquired.

Upon the completion of the above-described process, in step S84,implementation controlling unit 1313 completes image taking program 133and sound data recording program 136.

Then, in step S85, reproducing and editing engine program 131 receivesthe synthesized data, and script writing unit 1316 rewrites the scriptso that this data is displayed in the “Image” region between 0 and 8seconds.

At this point, the process of line 24 in the script shown in FIGS. 6 and7 is completed, and reproducing and editing processing of the newlygenerated script is implemented.

When reproducing and editing processing of the script shown in FIGS. 6and 7 is implemented in cellular phone 1 of the present datacommunication system, the above-described process is carried out oneline at a time.

Such reproducing and editing processing of the script is described belowusing the switching of concrete display screens shown in FIGS. 21, 22and 85. FIGS. 21 and 22 are diagrams sequentially showing the transitionof concrete display screens, and FIG. 85 shows concrete examples ofthese display screens in a single diagram.

FIG. 21(A) shows a concrete example of a screen that displays a list ofavailable templates. In the screen shown in FIG. 21(A), four templates“now I'm here,” “look at this!,” “considerable amount of free time” and“this tastes good!” are displayed so as to be selectable. Here, thoughin this example, only the names of the templates are displayed, it ismore preferable for concrete images for a message that can be preparedby utilizing a template to be displayed together with the titles asthumbnail mages.

In addition, though the templates that are displayed in a list aretemplates which are usually stored in a memory within the terminal,templates that can be downloaded from an external apparatus or a serverthrough a network connection may be displayed.

In addition, it is possible to display all the templates that have beenacquired or that can be acquired in the screen of this list oftemplates, or it is possible to display only some templates. In the casewhere such templates are displayed, a method for displaying only thetemplates which are often utilized and which are selected through useroperation or the like, a method for displaying only the templates whichhave been recently utilized, a method for displaying only templates fromwhich the terminal to which the message is to be transferred can beinterpreted or understood in the case where a message is transmitted toa person who has been designated in advance, and in the case where thefunctions of the terminal of the person to whom the message istransmitted are known and the like can be considered as a method forselecting which templates are displayed. As described above, in a casewhere a template is selected and displayed in accordance with thefunctions of the terminal of a person to whom a message is transmitted,a template for the purpose of taking a picture and transmitting thepicture on the spot by starting up a camera, for example, can be removedfrom the list, in the case where the terminal of a person to which amessage is transmitted is a cellular phone which does not have a camera.

Here, even in the case where only some templates are displayed in alist, as described above, it is preferable for all of the templateswhich have been acquired or which can be acquired to be displayed bymeans of a predetermined user operation. Furthermore, in the case wherethe functions of the terminal of a person to whom a message istransmitted can be found, colors or fonts are displayed in a differentmanner or an alarm screen is shown, in order to alert the user at aselected time, or the selection operation may be incapacitated for thetemplates where all of the described functions are found not to beinterpreted or implemented in the terminal of the person to whom amessage is transmitted.

Next, a process in a case where the template “considerable amount offree time” is selected in the screen shown in FIG. 21(A), and thereproduction process is implemented is concretely described.

With reference to FIG. 21, first, a predetermined navigation frame, thescript “considerable amount of free time” is selected, and thereproduction process is implemented. In cellular phone 1, theabove-described process is implemented and the script that is includedin this template is analyzed, and then, as shown in FIG. 21(B), theimplementation of this script is started.

A UI part for navigating the user, which is a UI part that changeschronologically as the script is implemented, is displayed on display144, as shown in the screens in and after FIG. 21(C). Concretelyspeaking, a letter sequence “startup of camera” is displayed on display144 at the time of reproduction of the second frame from among thescreens which are implemented and shown in FIG. 21(C). In the case wherethe user does not carry out any operation in this state, the lettersequence “startup of camera” is displayed for a predetermined period oftime, and after that, the script is implemented to the termination ofthe screen shown in FIG. 21(D), and the screen automatically returns tothe one at the beginning shown in FIG. 21(B), from where reproduction isrepeated.

Here, though a letter sequence is displayed as a UI part in thisexample, an icon image such as that of a camera or an animated image maybe displayed, or sound or music may be reproduced.

A user can carry out a predetermined key operation according to thetiming where the letter sequence “startup of camera” is displayed ondisplay 144, as shown in FIG. 21(C), and thereby, image taking program133 may be implemented so that a video can be taken by camera 149, asshown in FIGS. 21(E) and 21(F). Then, as shown in FIG. 21(G), thetemplate “considerable amount of free time” is rewritten so that thevideo data that was taken in the time slot when the letter sequence“startup of camera” was displayed can be reproduced. That is to say,when the implementation of the script is returned to the time where thestart of the script or the UI part of “startup of camera” is displayedon display 144 after the user has taken an image following thenavigation of the UI part, as shown in FIG. 22, the video that has beentaken as shown in FIG. 22(B) is reproduced in the time slot where the UIpart of “startup of camera” is displayed on display 144 when thetemplate “considerable amount of free time” was previously implemented.Here, in this case, it is preferable for the UI part of “startup ofcamera” not to be displayed in the time slot where the UI part of“startup of camera” is displayed when the video data is determined tohave been registered in this time slot at the time when the script isimplemented. Alternatively, it may be preferable for the display to bechanged to one where the UI part “retake video” is displayed in thistime slot, as shown in FIG. 22(B).

Hereinafter, implementation of the script that has automatically beenrewritten is repeated, until the operation of completion is carried out.

Next, “preview” is selected from the menu, and the process where thepreview is implemented is carried out. In the implementation of thepreview, as shown in FIG. 40, the script after editing is reproduced inthe state where the display and reproduction of the UI part are allincapacitated. That is to say, the letter sequence “retake video” shownin FIG. 22(B) is not displayed, and image taking program 133 is notexecuted, even in the case where the user carries out the predeterminedkey operation. Here, the process of the preview is described in thefollowing.

After the confirmation of the display of the script after editing in thepreview, “attachment to an e-mail” is selected from the menu, and thescript and media data are transmitted to another cellular phone 1. Here,the process of attachment to en e-mail is described in the following.

Furthermore, a case where the reproduction of the script shown in FIGS.29 and 30 is implemented in cellular phone 1 in the present datacommunication system is described as another concrete example of thepresent embodiment using the flowchart of FIGS. 31 and 32.

With reference to FIG. 31, when line 19 of the script shown in FIG. 30is implemented in cellular phone 1 according to the present embodiment,first, script analysis is carried out in syntax analyzer 1312 ofreproducing and editing engine program 131, and implementationcontrolling 1313 instructs image data displaying and reproducing program134 to display and reproduce image data “ProductImage.jpg” in “top”region between 0 and 10 seconds (S141). Then, image data displaying andreproducing program 134 displays and reproduces image data“ProductImage.jpg” on display 144 between 0 and 10 seconds.

Next, when line 20 of the script shown in FIG. 30 is implemented, instep S142, implementation controlling unit 1313 instructs, in the samemanner, text displaying reproducing program 134 to display and reproducetext data “ProductDesc.txt” in the “middle” region between 0 and 10seconds (S142). Then, text data displaying and reproducing program 138displays and reproduces text data “ProductDesc.txt” on display 144between 0 and 10 seconds.

Next, in the case where an event where a user operates the select keyoccurs between one and four seconds, the process shown in FIG. 32 whichis the process described in line 21 of the script shown in FIG. 30 isimplemented. That is to say, with reference to FIG. 32, when eventmanaging unit 1315 detects the occurrence of an event where the selectkey has been operated, implementation controlling unit 1313 refers tothe event processing timetable and determines the correspondingprocessing contents (action) (S151). Then, implementation controllingunit 1313 refers to the event processing timetable and starts up WWWbrowser program 151 which is the corresponding application (S152).

Next, WWW browser program 151 acquires the WWW contents, which are thendisplayed on display 144, via communication unit 110 (S153).

Upon the completion of the above-described process, implementationcontrolling unit 1313 completes WWW browser program 151 (S154).

At this point, the process of line 21 of the script shown in FIGS. 29and 30 is completed, and the process of reproduction of the script isimplemented.

When the process of reproduction of the script shown in FIGS. 29 and 30is implemented in cellular phone 1 of the present data communicationsystem, the above-described process is carried out one line at a time.Such a process of reproduction of the script is described using atransaction of concrete display screens shown in FIG. 33.

With reference to FIG. 33, cellular phone 1 of the present datacommunication system acquires a script from communication unit 110 andcarries out the process of reproduction. In cellular phone 1, theabove-described process is carried out so as to analyze the script, andimplementation of the script is started.

When implementation of the script is started, a UI part that changeschronologically as the script is implemented, which is a UI part thatnavigates the user, is displayed on display 144, as shown in FIG. 33.Concretely speaking, the icon image of “WWW browser” is displayed ondisplay 144 at the time of reproduction shown in FIG. 33(B). In the casewhere the user does not carry out any operation in this state, the stateautomatically transitions to that shown in FIG. 33(C), and the iconimage of “conversation” is displayed on display 144. Furthermore, in thecase where the user does not carry out any operation in this condition,the state automatically transitions to that shown in FIG. 33(D), and anicon image of “transmit e-mail” is displayed on display 144.

The user carries out a predetermined key operation in the time slotshown in FIG. 33(B), where the icon image of “WWW browser” is displayedon display 144, and thereby, WWW browser program 151 is executed, andpredetermined WWW contents can be acquired via communication unit 110and reproduced on display 144. When the WWW browser program is completedafter referring the WWW contents, the reproduction of the script isresumed from the point of the previous reproduction.

Hereinafter, the reproduction of the script is repeated until theoperation of completion is carried out.

Furthermore, the process of the preview in cellular phone 1 is describedusing the flowchart of FIG. 23 in the case where “preview” is selectedon the menu screen, of which a concrete example in FIG. 16. Withreference to FIG. 23, the process of the preview is approximately thesame as the reproducing and editing processing of the script shown inFIG. 9, and in the process of the preview, only the timetable for theprocess of media data drawing is prepared, in step S93, on the basis ofthe results of analysis in step S92 and on the basis of the drawinginformation describing portion of the script. Then, in step S94, thereproduction of the portion on the basis of the user navigationinformation describing portion of the script is not implemented, but theportion on the basis of the drawing information describing portion isimplemented. Then, implementation controlling unit 1313 implements theprocess that corresponds to this event (S96), when event managing unit1315 detects the occurrence of an event (YES in S95). Here, this processthat corresponds to the event is described in the following, by citing asubroutine. In the case of the process of the preview, the scriptdefinition inputting event is invalid, and only the system event andstable inputting event are processed in step S96.

At this point, the process of the preview of the script in cellularphone 1 is completed.

Next, the process that corresponds to the event which is implemented instep S96 is described using the flowchart of FIG. 38.

With reference to FIG. 38, first, when detection information of theoccurrence of an event is received from event managing unit 1315 in stepS95, implementation controlling unit 1312 pauses the script that isbeing implemented (S191).

Then, in the case where the event that has been detected in step S95 isan event that indicates that the implementation of the script hasreached the terminal of the implementation of this script (YES in S192),implementation controlling unit 1313 completes the implementation ofthis script (S1193), and the process returns to step S94 of FIG. 23 andthe implementation of the script is started again. That is to say, theimplementation of the script is returned to the beginning of theimplementation again, and the implementation of the script is repeatedfrom the beginning.

Meanwhile, in the case where the event that has been detected in stepS95 is an event other than the event that indicates that theimplementation of the script has reached the terminal of theimplementation (NO in S192), implementation controlling unit 1313determines the contents of processing on the basis of the event that hasbeen detected (S194).

In the case where the processing contents that have been determined instep S194 are “menu” (“process”=menu in S194), implementationcontrolling unit 1312 carries out the menu process shown in FIG. 15(S195). The procedure of implementation of the menu process is the sameas that in a case where the reproducing and editing is carried out, and,as shown in FIG. 39, the menu screen where the editing process becomeschoices instead of the preview process is exhibited on display 144. Inthe case where the editing process is selected, the reproducing andediting processing shown in FIG. 9 is implemented. Here, in the casewhere the reproducing and editing processing is started from the menu,it may be started at the start point of the script, or it may be startedat the time of implementation of the script at the time of the previewprocess.

Meanwhile, in the case where the processing contents which have beendetermined in step S194 are “skip” (“process”=skip in S194),implementation controlling unit 1312 refers to the media drawingprocessing timetable so as to determine the media drawing process thatis to be implemented next and updates the implementation of the scriptto the starting time of this media drawing process (S196).

Here, in addition to the above-described starting time of the mediadrawing process that is to be implemented next, the starting time of themedia drawing process that is to be implemented as any of the processesafter the next or the time of implementation when a predetermined timehas passed in the case where the script is continued to be implementedas it is without a selection of the skipping process may be the time towhich the implementation of the script is moved.

Next, in the case where a process of attaching this script and mediadata to an e-mail is selected in the menu screen of which a concreteexample is shown in FIG. 16, the process of attachment to an e-mail incellular phone 1 is described using the flowchart of FIG. 34.

With reference to FIG. 34, first, implementation controlling unit 1313displays a screen for selecting whether or not the script is transmittedas multimedia data specified for reproduction on display 144. Then, inthe case where attachment of the multimedia data specified forreproduction is not selected on this screen (NO in S161), implementationcontrolling unit 1313 sets a suggestion as to how the multimedia datacan be utilized in another cellular phone 1 that receives the multimediadata (S165). That is to say, in step S165, implementation controllingunit 1313 sets whether this multimedia data is to be utilized as atemplate for navigating the contents editing operation or as thecontents simply for reference. Then, the process is made to progress tostep S163.

Meanwhile, in the case where the attachment of the multimedia dataspecified for reproduction is selected on the screen for selectingwhether or not the script is transmitted as multimedia data specifiedfor reproduction (YES in S161), implementation controlling unit 1313deletes the user navigation information describing portion (S162).

Next, implementation controlling unit 1313 writes a script in memoryunit 130 (S163). Then, implementation controlling unit 1313 sets theabove-described script and multimedia data as a file to be attached, andstarts up an e-mail program (S164).

Here, in the case where the attachment of the multimedia data specifiedfor reproduction is selected in the above-described process forattachment to an e-mail (YES in S161), the multimedia data templatebefore being edited may be added to an e-mail together with themultimedia data specified for reproduction. In addition, in the casewhere there is a portion of description where it has been found inadvance that cellular phone 1 that receives the e-mail cannot processthe portion of description, in step S162, which is the above-describedprocess for attachment to an e-mail, such a portion of description maybe deleted from the script.

Second Embodiment

Next, with reference to the drawings, the second embodiment of thepresent invention is described.

The configuration of the data communication system that includes amultimedia data processing device according to the second embodiment ofthe present invention is the same as in the case of the first embodimentshown in FIG. 1. In addition, the configuration of cellular phones 1 andthe programs stored in memory unit 130 and the flow of data processingof a cellular phone 1 are also the same as in the case of the firstembodiment shown in FIGS. 2 and 3, respectively.

First, a case is described where a script that includes reproductionorder information of media data is reproduced in cellular phone 1, whichis a multimedia processing device, according to the second embodiment,and a script that has been modified when media data is written into thescript, and that includes the reproduction order information of themedia data is reproduced.

The template shown in FIG. 41A is formed of three scenes, scene 1 toscene 3. The transition between these scenes is not particularly limitedaccording to the present invention, though a case where time informationis described in the script within the template and the transitionfollows this, a case where the transition follows the time of switchingthat has been set in the terminal, a case where a user clarifies theswitching of the scenes and the like are considered. The concreteexample shown in FIGS. 41A and 41B is a case where the still image ofscene 2 shown in FIG. 41A (B) and the text of scene 3 shown in FIG. 41A(C) are switched by the user in a manner as shown in FIGS. 41B (E) and41B (F), respectively, and thereby, modification is carried out.

Furthermore, FIG. 47 shows a concrete example of a script forimplementing the screen display of FIG. 41A. The script shown in FIG. 47is a script that represents a sequential reproduction of the scenes thatexpress collected media data that is reproduced in parallel. That is tosay, with reference to FIG. 47, descriptions in lines 15 to 17, lines 18to 21 and lines 22 to 25 represent the respective scenes 1 to 3.

In further detail, line 19 describes a reproduction of an image, andline 20 describes the reproduction of a text in scene 2 that isdescribed in lines 18 to 21. These images and texts belong to the scenedescribed in lines 18 to 21, and are reproduced in parallel.

Furthermore, the time that is displayed after “dur=” within the scriptrepresents the time for reproducing the corresponding scene, and afterthis time has passed, the reproduction switches to the next scene.Though a script is shown as one where such a describing method is usedin the present embodiment, the method for describing a script is notlimited in the present invention, and any style of script may be used,as long as sequence information for reproducing media can be described.

Here, sequence indicates a relationship of chronological order at thetime of reproduction of multimedia data. A sequence includes one or morespecific sections along the axis of the reproduction time as elements(concretely speaking, “scene,” “slide,” and more concretely, segmentsdefined by being surrounded by <par> tags or <seq> tags in SMIL). Thesequence is managed by grasping the relationship of the order of therespective segments along the time axis, and by preparing andmaintaining the data structure that becomes a clue at the time when thereproduction of (media data that is included in) the respective segmentsis started or stopped, or loading and unloading of media data is carriedout. Concretely speaking, a timetable as shown in FIGS. 10 to 12 isused, and thereby, the sequence is managed. The times when therespective segments are started and completed are referred to aschanging points of a sequence along the axis of reproduction time. Inthe concrete example of the UI allocation state displaying timetableshown in FIG. 7, 0^(th), 8^(th), 15^(th), 19^(th) seconds correspond tothe changing points of the sequence.

Furthermore, FIG. 42 is a block diagram showing the functionconfiguration of reproducing and editing engine program 131 of cellularphone 1 that is a multimedia data processing device according to thepresent embodiment. Control unit 120 executes reproducing and editingengine program 131 that is stored in memory unit 130, and thereby, therespective functions shown in FIG. 42 are implemented in cellular phone1.

With reference to FIG. 42, the functions of reproducing and editingengine program 131 of cellular phone 1 according to the presentembodiment is formed so as to include a template storing unit 1000, aselecting unit 1001, a script and data temporarily storing unit 1002, ascript implementing unit 1003, a user changed data determining unit1004, a terminal function selecting unit 1005, a terminal functionimplementing unit 1006, a data changing unit 1007, a script and datachange confirming unit 1008, a script and data outputting unit 1009, anoutput script and data storing unit 1010 and a data burying unit 1011.

Template storing unit 1000 stores one or more template that is passed toand received by a receiving unit 1015. Though a template that is storedin template storing unit 1000 usually includes both a script as shown inFIG. 47 and sample media data (still image or text) that is utilized bythis script, the script may be in another form, in the case where onlythe media data is in the form of a file or the like, so as to beseparately managed, exists in a server that is connected to a network,or is accessible when the template is utilized.

Here, it is preferable for operations such as separately transferring orstoring only the media data that is included in the template and that istaken out from the template to be prohibited or permitted from beingcarried out according to the description of the script that is includedin the template. Furthermore, it is preferable for two or more templatesto be usually stored in a terminal in a manner where a template that isdesired to be used by a user can be added from a server that isconnected to a network from a memory card or the like.

Selecting unit 1001 receives a selection of a template from a user inthe list of templates that are displayed so as to be selectable. Thetemplate selection screen of FIG. 21 is a typical example.

Script and data temporarily storing unit 1002 temporarily stores ascript and media data which are included in a template. A scriptincludes a portion that is rewritable, and therefore, a template isusually copied and stored in script and data temporarily storing unit1002, which is a temporary storing area for this. In the case wheremedia data itself is not modified, for example, when rewriting of thefile name of media data, however, it is not necessary to especiallyprepare duplication.

Script implementing unit 1003 is a portion of the core of the presentinvention. According to the present invention, the script is used toimplement a display of which the scene has been switched. It ispreferable for script implementing unit 1003 to be managed by preparingin advance a table for a scene or time at the time of scriptreproduction

User changed data determining unit 1004 determines whether or not amedium that is included in a template can be changed. The majority of amedia that is included in a template cannot be rewritten, but referenceinformation within the media for referring to a media, such as a mediafile name is rewritten, and thereby, multimedia data that is desired bythe user can be finally generated.

Media data that is included in a template is usually described in ascript by being designated the form of a file name. User changed datadetermining unit 1004 determines whether or not this media data can bechanged. In the case of scene 2 that is shown in FIG. 41A (B), forexample, it is possible to switch still images on the terminal side, andtherefore, user changed data determining unit 1004 determines thatswitching of still images is possible. In addition, even in the casewhere a still image file is designated in a script, there may be a casewhere data change is impossible in this still image file.

Furthermore, in the case of a script that is managed by time, it ispreferable to make it possible for the time when this media data isreproduced to be selected in user changed data determining unit 1004 sothat the reproduction time itself is rewritten or a fourth scene isadded to the three scenes, after these are determined by user changeddata determining unit 1004.

Terminal function selecting unit 1005 selects a terminal function thatis required to change the portion that has been selected by user changeddata determining unit 1004 from among the functions provided with thiscellular phone 1. In the case where the portion selected by user changeddata determining unit 1004 is a still image, for example, terminalfunction selecting unit 1005 selects the function for switching stillimages. In addition, in the case where this cellular phone 1 is aterminal with a camera, for example, it is preferable for terminalfunction selecting unit 1005 to be able to select the function ofstarting up a camera.

Terminal function implementing unit 1006 acquires media data by using aterminal function that has been selected by terminal function selectingunit 1005. Then terminal function implementing unit 1006 outputs theacquired media data to data changing unit 1007.

Data changing unit 1007 switches the media data that has been inputtedfrom terminal function implementing unit 1006 to the original media datathat is referred from the script. At this time, it is possible for astill image in a template to be converted to another type of media data,such as video or text, after the change.

In addition, at this time, in the case where reference information tomedia data in a script is information that is described under a filename, the media data after the change is maintained in a file style, andthis name is used as a file name that is described in the script, or inthe case where the media data after the change is originally in a filestyle and is managed by having a name attached to it, it is necessary torewrite the portion within the script that designates this media data tothis file name.

Script and data change confirming unit 1008 receives confirmation thatchanges in the script and media data have all been completed from theuser. Then, script and data change confirming unit 1008 outputs theconfirmation of change in the script and media data to script and dataoutputting unit 1009.

Script and data outputting unit 1009 outputs the script and media datawhich have been changed and confirmed. Script and data outputting unit1009 transmits data to another terminal or a server typically in theform of an e-mail such as an MMS (Multimedia Messaging Service). Then,script and data outputting unit 1009 outputs the script and media datathat have been outputted to another terminal or the like to outputscript and data storing unit 1010 and data burying unit 1011.

Output script and data storing unit 1010 stores the script and mediadata that have been outputted from script and data outputting unit 1009.In addition, data burying unit 1011 takes out media data from theoutputted script and media data, and buries it in a template.

Next, processing in the reproducing and editing engine program ofcellular phone 1 that is a multimedia data processing device accordingto the second embodiment is described using the flowchart of FIG. 43.The processing shown in the flowchart of FIG. 43 is also implemented bycontrol unit 120 of cellular phone 1, reading out reproducing andediting engine program 131 of which the configuration is shown in FIG.42 from memory unit 130, and implementing it.

With reference to FIG. 43, first, selecting unit 1001 reads out atemplate that is stored in template storing unit 1000 or a template thatis acquired from another device, and displays a list from which atemplate can be selected on display 144, and then, accepts a selectionby a user (S1100). It is preferable for selecting unit 1001 to convertthe initial or representative screen of each template into a thumbnail,and to display this together with the title of this template in stepS1100.

Next, Script implementing unit 1003 implements reproducing and editingprocessing where editing is carried out in accordance with theinstruction by the user while reproducing a template that is selected instep S1100 (S1101). Here, the reproducing and editing processing in stepS1101 is described in detail below. Furthermore, though reproduction andediting generally have modes that can be switched by the user, it ispreferable for editing and reproduction to be carried outsimultaneously.

Next, script and data change confirming unit 1008 confirms whether ornot change of script by the user has been completed (S1102). Then, inthe case where the change of the script has been completed (YES inS1102), script and data outputting unit 1009 outputs either or both ofthe confirmed script and data (S1103).

At this point, reproducing and editing processing of a templateaccording to the present embodiment is completed.

Next, the reproducing and editing processing in step S1101 is describedusing the flowchart of FIG. 46.

With reference to FIG. 46, first, script implementing unit 1003 switchesthe reproduced scene in accordance with the script (S1200). Though theprocess in step S1200 is not limited in the present invention, a methodfor first preparing a timetable or a switching table as shown in FIG. 58and for shifting to the next scene on the basis of this can be cited asthe typical method for processing

FIG. 58 shows a concrete example of a media managing table in the casewhere the script shown in FIG. 47 is reproduced and edited. Concretelyspeaking, with reference to FIG. 58, “still image 1” is reproduced in“scene 1” (FIG. 58(A)), “still image 2” and “text 2” are reproduced in“scene 2” (FIG. 58(B)), and “still image 3” and “text 3” are reproducedin “scene 3” (FIG. 58(C)). Data changing unit 1007 sequentiallyreproduces and edits scene units with reference to such a media managingtable in step S1200.

Next, script and data outputting unit 1009 outputs data using a functionof a terminal (S1201). The data form outputted in step S1201 may be anyof still images, animated images, music, text and the like. Here, theoutput processing in step S1201 is described in detail below, by citinga subroutine.

Next, script implementing unit 1003 determines whether the mode is amodification mode or a reproduction mode. In the case where a newtemplate is prepared on the basis of an existing template, sometimes,switching to a reproduction mode can be made unnecessary simply byimplementing a modification mode. Contrarily, in the case where ane-mail or a message that has been sent from another terminal is utilizedas a new template, the mode is usually a reproduction mode, and it ispreferable for the mode to be changed to a modification mode.

In the case where the mode is a modification mode in step S1202 (YES inS1202), an input is additionally acquired from the user, and userchanged data determining unit 1004 determines the type of input (S1203).

As a result of the determination in step S1203, in the case where theinput from the user is an input that indicates a reproduction of theprevious scene (“previous scene” in S1203), script implementing unit1003 carries out a process for displaying the scene before this script(S1204).

Alternatively, as a result of the determination in step S1203, in thecase where the input from the user is an input that indicates a processof changing the script (“changing process” in S1203), data changing unit1007 carries out a concrete changing task (S1205). Here, the changingtask in step S1205 is described in detail below, by citing a subroutine.

Furthermore, script and data change confirming unit 1008 determineswhether or not the script is again confirmed after obtaining an inputfrom the user after the change in step S1205 (S1205). As a result of thedetermination, in step S1206, in the case where the script after thechange is reconfirmed (“second output” in S1206), the process returns tostep S1201, and script and data outputting unit 1009 outputs the dataafter the same scene has been changed in step S1201. This is effectivewhen the scene cannot be confirmed merely by changing the still image,such as in a case where a still image and music are simultaneouslyoutputted.

Then, this reproducing and editing processing is completed in the casewhere the mode is not a modification mode in step S1202 (NO in S1202),and the reproduction of this script has been completed (YES in S1207).

Next, the data outputting process in step S1201 is described in detailusing the flowchart of FIG. 44.

With reference to FIG. 44, first, user changed data determining unit1004 determines the existence of changed data within this scene (S1110).The determination process in step S1110 is described in detail below, byciting a subroutine.

Next, terminal function selecting unit 1005 selects a means for changingthe changed data that has been determined in step S1110 (S1111). Thesame script may have a separate function that can be used forreproduction, for example, depending on the terminal, as shown in FIG.49. It is possible to take a difference in software of the terminal inaddition to a difference in the hardware of the terminal intoconsideration for the selection of a function. In this case, it ispreferable to select a function that is available for each terminal,such as selection of a still image file or selection of a video camera,in step 1111. Here, the process of selecting a means for changing thechanged data in step S1111 is described in detail below, by citing asubroutine.

Next, data changing unit 1007 displays guidance for change on display144 as an operation guide (S1112). The modification guidance displayedin step S1112 is a guide function for selecting a file to be switched inthe case where, for example, a still image is switched, and typicallyindicates a display of a still image file selection button. Datachanging unit 1007 dynamically generates such guidance in step S1112,and thereby, the user can be easily notified as to how the data can bechanged.

Next, script implementing unit 1003 implements outputting of a scene,together with the change guidance displayed in step S1112 (S1113).

Then, script and data temporarily storing unit 1002 stores a pair of themeans (application) for modification selected in step S1111 and theinput event that corresponds to the modification guidance that isgenerated in step S1112 in a scene event table for each piece ofmodification data (S1114). In step S1114, concretely speaking, scriptand data temporarily storing unit 1002 stores, as shown in the concreteexample of the scene event table of FIG. 64, that changed data “stillimage 2” can be changed by means of an “image taking” application thatis started up by an “F1” key event, and by means of a “file acquiring”application that is started up by an “F2” key event, and changed data“text 2” can be changed by means of a “file acquiring” application thatis started up by a “select” key event.

Next, the process of changing in step S1205 is described using theflowchart of FIG. 65.

With reference to FIG. 65, first, data changing unit 1007 refers to ascene event table (S1501). In the case where the user inputted eventthat has been inputted in step S1203 is not registered in the sceneevent table (NO in S1502), this process is completed.

In the case where the user inputted event that has been inputted in stepS1203 is registered in the scene event table (YES in S1502), scriptimplementing unit 1003 determines and starts up the correspondingapplication selected from the scene event table (S1503). Then, datachanging unit 1007 determines the corresponding user changed dataselected from the scene event table and acquires media data from theapplication that has been started up, and then, this user changed datais rewritten to the media data that has been acquired from theapplication that has been started up (S1504). Concretely speaking, whenkey “F1” is pressed, an “image taking” application is started up in stepS 1503. In addition, when the user takes a still image using the “imagetaking” application and finishes this application, user changed data“still image 2” is rewritten with the newly taken data in step S1504.

Here, in the case where two or more places into which media data thathas been inputted into a template can be inserted exist when the usercarries out an inputting operation of media data, such as images, sound,video and text, in the above-described example, it is possible to insertthis media data in two or more places. Alternatively, the media datathat has been inputted by the user may be divided so that a portionthereof is inserted in two or more places.

In addition, in the case where media data having a time axis, such assound, music, video and animation, is inserted into a template, thismedia data may be prohibited from being inserted, reproduction speed ofeach piece of media data may be increased so as to adjust thereproducing time to a prescribed time, reproduction may be completed onthe way after a prescribed time has passed, or only a portion that iscontained within a prescribed time may be reproduced, in the case wherethe reproduction time of the respective pieces of media data is longerthan the time that has been prescribed in advance in the template.Alternatively, time prescription of the template may be changed so thateach piece of media data is completely reproduced.

In addition, when media data such as text, still images, animatedimages, sound, music and animation is inserted, file selection andstartup of an application, such as a camera, are carried out, inaccordance with the inserted media data in the above-described example.However, a user may select media data freely and convert this media datato media data that is contained in a template. Extraction of only soundfrom a video that has been taken so as to be buried in a template,sampling of the leading frame of a video as a still image so as to beburied in a template or the like, for example, may be carried out.

Here, at the time of the above-described text inputting, it ispreferable to read in sentence formats stored in the apparatus, textwhich is temporarily stored in a clipboard, an address book or schedulethat is stored within an apparatus and that can be acquired as textdata, and data of a transmitted or received e-mail.

Next, the determination process in step S1110 is described using theflowchart of FIG. 59.

With reference to FIG. 59, user changed data determining unit 1004refers to the scene that is reproduced at this time and the mediamanaging table (S1301 and S1302), and determines the media data of thisscene. In the case where “scene 2” of the script shown in FIG. 47 isreproduced in cellular phone 1 at this time, user changed datadetermining unit 1004 refers to the media managing table shown in FIG.58 and determines “still image 2” and “text 2” to be changed data.

Next, the process of selecting a means for changing of the changed datain step S1111 is described with reference to FIG. 60. FIG. 60 is a tablefor recording the correspondence between data acquiring programs storedin cellular phone 1 and media types supported by cellular phone 1, andis a table that is stored in memory unit 130 of cellular phone 1.Concretely speaking, the table shown in FIG. 60 shows that an “imagetaking” application is registered in this cellular phone 1 so that“still images” and “videos” can be acquired, a “sound data recording”application is registered so that “sound” can be acquired, and a “fileacquiring” application is registered so that “still images,” “videos”and “text” can be acquired.

In step S1111, terminal function selecting unit 1005 refers to theabove-described table and selects an available means for changing on thebasis of the media type of the changed data that has been determined instep S1110. In the case where the media type of the changed data is“still images,” for example, the applications of “image taking” and“file acquiring” are selected.

FIG. 53 shows concrete examples of displays where screens transition atthe time when a script is implemented in cellular phone 1. Withreference to FIG. 53, in scene 2, the process in step S1113 isimplemented so that still images and texts are displayed, and at thesame time, the process in step S1112 is implemented so that changingguidance is displayed. That is to say, text (guidance) “you can attach apicture here” are displayed in scene 2 of FIG. 53, and when thecorresponding button is pressed, the data of still images in a datafolder is displayed in a list in the form of thumbnails. In this screendisplaying a list, it is more preferable for file names and data sizeinformation to be shown together.

In addition, in the case where two or more types of media, such as stillimages and music, are outputted in the same scene, the present cellularphone 1 may have a configuration where any of the following methods isused: a method for selecting either still image data input or music datainput by means of a user operation, such as the pressing of a shortcutkey, from the display where two or more modification guidance are shown,for example, “you can attach a picture here” and “you can attach musichere;” a method for showing a first guidance of “you can attach apicture here” followed by a guidance “you can attach music here” bychronologically switching the guidance; or a method for shifting to astill image data input or a music data input as a result of a selectionby a user from a menu that is displayed so as to be selectable, in sucha manner that the guidance “you can attach data here” showing that atype of media data can be inserted is displayed and transitions to thenext screen when the corresponding button is pressed, where “attach apicture” and “attach music” are displayed in the next screen.

In addition, in the case where media data that can be modified and mediadata that cannot be modified can be distinguished, it is preferable forthe present cellular phone 1 to show the user the data that can bemodified in this scene, by distinguishing it from the others. Suchdetermination can be made by describing whether or not media data can bechanged in the script.

More preferably, the present cellular phone 1 is provided with afunction of skipping to the next scene or returning to the previousscene. Efficient preparation and confirmation of an e-mail becomepossible during conventional reproduction, in a manner where a user canpress a button to skip the remaining seven seconds after one second haspassed by using a “next scene” function in the case where scene 1 isdisplayed for 8 seconds.

More preferably, the present cellular phone 1 has a function of stoppingthe reproduction at a point in time when a user shows an intention toedit. It is preferable for a user to press a button, for example, whenstill images are switched in scene 2 of FIG. 53, and thereby, it ispreferable for the timer of reproduction to be stopped when a stillimage is selected from a still image selecting screen. Here, the startof the scene at the time of reproduction when the scene is stopped forcarrying out an editing operation, reproduction from the beginning ofthe entirety and resumption from the beginning of this scene can beconsidered in the case where the reproduction is resumed after changingof data has been completed. The beginning of this scene indicates thebeginning of scene 2, that is to say, the 8^(th) second, in the casewhere the data is switched by pressing an editing button at the 13^(th)second, where scene 2 is planned to be reproduced from the 8^(th) secondto the 15 second.

Here, a case is considered in the above-described editing process of atemplate, where a user wants to change the editing of a template toediting of a template that is different from the template that isselected at this time by the user and is being edited at a point in timewhere media data, such as text, still images, video, sound, music andanimation is inserted in one or more places of the template. In such acase, the media data that has been inserted so far may be discarded orinserted into a template after the change to whatever template theediting is changed. In addition, in such a case, it may be impossible toinsert the media data that has been inserted before the change into atemplate after the change, and at this time, the change to such atemplate may not be permitted, or the media data that is impossible tobe inserted may be discarded so that only the media data that can beinserted is inserted. In addition, in the case where the media data thathas been inserted into the template before the change can be inserted intwo or more places of the template after the change, the media data maybe inserted into one place, or two or more places. In addition, from thestart, the change to another template may be completely prohibited.

Modification

Next, a process where a script that includes reproduction orderinformation of media data and implementation order information of theactions is reproduced so that the actions are implemented and the scriptthat includes the reproduction order information of media data isprepared is described.

The description of the script that is shown in FIG. 50 defines theactions and drawing regions in the same manner as the description of thescript shown in FIG. 6. The description of the script shown in FIG. 61defines the reproduction of the media data and implementation orderinformation of the actions in lines 17 to 33. In addition, lines 18 to32 show sequential implementations of scene 1 to scene 3, which arerespectively defined in lines 19 to 21, lines 22 to 26 and lines 27 to31.

In addition, lines 22 to 26 define the implementation of the secondscene. In further detail, line 23 describes that the image shown by“SecondImage.jpg” is displayed in the “Image” region. Line 24 describesthat the text shown by “SecondText.txt” is displayed in the “Text”region. Line 25 describes that the action that is referred by“act_image” is implemented in the case where a “select” key event of“UI” is detected.

Furthermore, FIG. 48 is a block diagram showing a functionalconfiguration of reproducing and editing engine program 131 of cellularphone 1 according to a modification of the second embodiment. In FIG.48, the components to which the same numbers as in FIG. 42 are attachedare the same components as those shown in FIG. 42. That is to say, withreference to FIG. 48, the functions of reproducing and editing engineprogram 131 of cellular phone 1 according to the modification of thesecond embodiment are formed so as to include a template storing unit1000, a selecting unit 1001, a script and data temporarily storing unit1002, a script implementing unit 1003, a terminal function selectingunit 1012, an action determining unit 1013, an action implementing unit1014, a data change confirming unit 1008, a script and data outputtingunit 1009, an output script and data storing unit 1010 and a databurying unit 1111. Here, components which are different from those ofFIG. 42 are described.

Terminal function selecting unit 1012 selects a terminal function inaccordance with the description of the script. A terminal function isdescribed in a script in a form such as, for example, “dest” in the“action” sentence of lines 12 to 15 of FIG. 50. Then, terminal functionselecting unit 1012 outputs this result of selection to actionimplementing unit 1014.

Action determining unit 1013 determines the actions in accordance withthe description of the script. The determination of the actions inaction determining unit 1013 is, for example, implemented byinterpreting the description with “type” in lines 12 to 15 of the scriptshown in FIG. 50. Then, action determining unit 1013 outputs thisdetermination result to action implementing unit 1014.

Action implementing unit 1014 processes the data that has been obtainedby using the function selected by terminal function selecting unit 1012on the basis of the actions determined by action determining unit 1013.It becomes possible to carry out a variety of processes, such assynthesis of images, connection of music and special effects, inaddition to simple insertions and replacements, in action implementingunit 1014, by implementing the actions which are clearly described.

Here, the process in the reproducing and editing engine program incellular phone 1 according to the modification of the second embodimentis the same as that shown in FIG. 43. In addition, the reproducing andediting processing of the navigation frame which is a script in cellularphone 1 according to the modification of the second embodiment is alsoapproximately the same as that shown in FIG. 46.

Here, in the present modification, first, an action managing table asshown in FIG. 63 is prepared in the processing shown in FIG. 46, inaddition to a media managing table as shown in FIG. 58. FIG. 63 is adiagram showing a concrete example of an action managing table in thecase where the script shown in FIG. 61 is reproduced and edited.Concretely speaking, with reference to FIG. 63, there is no action thatcan be implemented in “scene 1” (FIG. 63(A)), a “still image buryingaction” can be implemented in “scene 2” (FIG. 63(B)), and a “textburying action” can be implemented in “scene 3” (FIG. 63(C)). Cellularphone 1 in the modification refers to such an action managing table sothat scene units are sequentially reproduced and edited.

Next, the data outputting process in step 1201 according to themodification is described in detail using the flowchart of FIG. 62.

With reference to FIG. 62, first, action determining unit 1013 refers tothe action managing table so as to determine the actions that can beimplemented in this scene (S1401). Next, action determining unit 1013displays guidance for action implementation on display 144 (S1402). Theguidance for action implementation is a guide function of the operationfor starting the implementation of the actions. The guidance displayingprocess of action implementation in step S1402 is the same as thedisplay process of changing guidance in step S1112 according to thefirst embodiment. Then, Script implementing unit 1003 implements theoutput of the scene together with the guidance displayed in step S1402(S1403).

FIG. 45 shows a concrete example of a display of a screen transition atthe time when a script is implemented in cellular phone 1 according tothe modification. With reference to FIG. 45, the process in step S1113is implemented so that a still image is displayed, and at the same time,the process in step S1112 is implemented, so that the changing guidanceis displayed in scene 2 shown in FIG. 45(B). That is to say, in scene 2shown in FIG. 45(B), text (guidance) “you can attach a picture here” aredisplayed, and a camera application is started up, as shown in FIG.45(D) when the corresponding button is pressed.

Here, the descriptions specific to the respective terminals havingdifferent available functions, as shown in FIG. 49, may all be describedin the script, and only the descriptions of the functions that can beused by the terminal at the time of the reproduction of the template maybe made effective. 3D liquid crystal display function, display functionwith two or more screens and the like, for example, can be speciallycited as the above-described functions.

In addition, reference information to media data having differentcontents for each attribute of the user who is an object may all bedescribed in the script, and only the media data that is suitable forthe attributes of the user who is an object may be made effective at thetime of the utilization of the template in the configuration. Thelanguages utilized by a user, regions where a user lives and the like,for example, can be specially cited as the above-described attributes.

Here, though in the above embodiments, a method for displaying a UI partthat is indicates the startup of a camera on a screen and for startingup a camera by means of a predetermined key operation is described, asshown in FIG. 52, the camera may be started up in an in-line manner inthis time slot, a UI part that indicates that an image is taken may bedisplayed for a predetermined period of time (FIG. 52(D)), an image maybe taken by the camera when a predetermined key operation is carried outwithin this time slot, and this image or video may be reproduced on thespot. Here, at this time, in the case where a predetermined time haspassed during which the UI part of “take image” is displayed withouttaking an image, the next reproduction process may be started in thistime slot. Alternatively, the reproduction may be paused at the time ofthe termination of this time slot, and the system may stand by until the“take image” operation is carried out. Alternatively, the reproductionin this time slot may be repeated by returning to the beginning of thistime slot. This repeating process may be carried out infinitely, as longas the “take image” operation is not carried out or the interruptionprocess is not carried out, or the process may be repeated only for apredetermined number of times, after which the process may proceed tothe next process, even in the case where the “take image” operation isnot finished.

Here, a case is considered where the total size of the message dataexceeds the upper limit of the size that can be transmitted in theprocess for preparing a message in the above-described embodiment. As ameasure against this, an alarm of an excessive size may be issued so asto halt the transmission at the point in time when the user carries outan operation of transmitting the message. Alternatively, the sum of thesize may be searched each time that the user inserts media data, so thatan alarm is issued at the point where the size exceeds the limit, andthen, this insertion operation may be invalidated. Alternatively, in thecase where the size of the media data which is to be inserted is foundin advance, it may be determined whether or not the sum of the totalmessage size at this time and the size of such media data exceeds theupper limit of the size that can be transmitted, and the selection forthe media data in the case where the sum exceeds the limit may not bepermitted.

Here, the above-described FIGS. 50 and 61 show a concrete example of thescript which is used only on the preparing side. Lines 12 to 15 of thisscript are solely utilized on the device side that prepares andtransmits a message. Other portions are required at the time when ascript is reproduced on the device side that receives the message. Whensuch a script is outputted in script and data outputting unit 1009,unnecessary descriptions in the receiving device, the commands thatcannot be transmitted in the communication procedure and the like whichare included in the script should be finally eliminated or deletedbefore being outputted. In such a case, it is preferable to delete thesein script and data outputting unit 1009 which is provided with rules tothe above-described effect concerning deletion.

Here, it is more preferable for script and data outputting unit 1009 toadd the address from which the template that is utilized at this timehas been downloaded before sending a message, so that it becomesconvenient in the case where the person who receives the messagedownloads the template itself from a server on the reception side forthe purpose of preparing another message by utilizing the same template.

Furthermore, it is preferable to convert a script into a format such as,for example, HTML, in script in data processing unit 1009, so that theconverted script can be transmitted with the original script. By doingthis, it becomes possible to reproduce the script on the reception sidein the case where the original script cannot be processed on thereceiving side but it is possible the process the converted script.

Furthermore, the script before the deletion process is carried out, andmedia data may be stored in the output script and data storing unit 1010so that it becomes convenient in the case where the template is editedso as to become the original, and the transmitted message is editedagain so as to be transmitted. In addition, for the same purpose, asshown in FIG. 55, a table that shows the correspondence between thetransmitted message an the utilized template may be maintained, and, forexample, the above-described table may be searched so as toautomatically select template “considerable amount of free time,” andthe media data that is included in message “1221” may be buried intemplate “considerable amount of free time” by data burying unit 1011,and after that, editing may be carried out in the case where message“1221” is desired to be edited again.

In order to make editing easier, as shown in FIGS. 51A to 51D, it ispreferable to prepare thumbnail for each scene. This can be easilyimplemented by preparing the entirety of the timetable or the sequencetable, as shown in FIG. 10, by interpreting the entirety of the scriptthat is included in the template. By preparing such a thumbnail for eachscene, the reproduction of this scene can be started at the point ofreproduction implementation triggered by the selection of an arbitrarythumbnail by means of a user operation from a list of thumbnails that isdisplayed on the basis of the script, and then, in the case where thescript is rewritten, it becomes easily possible to rewrite only desireddata without reproducing this script from the beginning.

Such a script is implemented in cellular phone 1, and thereby, thecontents of multimedia can be easily prepared. That is to say, apredetermined operation can be carried out following the guidanceinformation which chronologically varies and is displayed when thisscript is carried out, and thereby, multimedia contents can be easilyprepared according to an instinctive method. In addition, apredetermined operation can be carried out on a template into whichmedia data has already been inserted by following guidance whichchronologically varies and is displayed when this script is carried out,and thereby, media data can be easily replaced and multimedia contentsthat are desired by the user can be easily prepared.

Furthermore, such a template is transmitted to another cellular phone 1so as to request a reply, and thereby, it is possible to make anoperation desired by the sender side be carried out on the receiverside. A text message to the effect that, for example, the insertion of anew video to template “considerable amount of free time” as shown inFIG. 21 is requested is added to an e-mail which is transmitted toanother cellular phone 1, and thereby, implementation of an editing taskwhere new video data is inserted into this template can be instructed tothe user of this cellular phone 1.

Third Embodiment

Next, a case where the multimedia data processing device is a televisionreceiver (hereinafter abbreviated to TV) is described as the thirdembodiment. That is to say, a layout and time information of atelevision screen are described in a template according to the thirdembodiment, and a television program is handled as one piece of mediadata.

With reference to FIG. 74, a data communication system is formed so asto include a TV 6 that is a multimedia processing device, a templateproviding server 2 which provides a template that is described in ascript language, an EPG server 3 which provides program information andprogram related information of a TV, a WWW server 4 which providesinternet contents, and a broadcasting station 5 which provides a programto TV 6 via broadcasting waves.

The configuration of TV 6 as a multimedia data processing device is thesame as that shown in FIG. 4 according to the first embodiment.Furthermore, TV 6 includes main region 7401 and sub-regions 7402 and7203 in its display unit.

FIG. 75 shows a concrete example of a script that includes a drawinginformation describing portion that refers to media data, and FIG. 76shows a concrete example of a script that includes a program tableinformation describing portion that describes program information inaccordance with a time slot, and FIG. 77 shows a concrete example of ascript that includes a description which relates the two scripts shownin FIGS. 75 and 76 to each other. Here, the script shown in FIG. 76corresponds to the script that includes the user navigation informationdescribing portion according to the first embodiment, and is a scriptwhich does not navigate the operation of the user who views and listensto TV 6, but works to automatically determine the program to bedisplayed and reproduced in accordance with a time slot according to thepresent embodiment. Program table information is obtained by analyzingthis script.

Next, the flow of data processing in TV 6 is shown. The flow of readingin and implementing processing of a script and event processing isroughly the same as that of the processing shown in FIG. 9, andtherefore, the flow is described with reference to FIG. 9.

With reference to FIG. 9, first, TV 6 acquires the respective scriptsshown in FIGS. 75, 76 and 77 (S12) and reads in these scripts (S13).Furthermore, the syntaxes of the respective scripts that have been readin are analyzed (S14) and a timetable is prepared (S15). After that,these scripts are implemented (S16) in accordance with the preparedtimetable.

Concretely speaking, lines 14 and 15 of FIG. 75 designate an image fileon the WWW server, and therefore, the corresponding file is acquiredfrom WWW server 4 in accordance with such a description in step S12, andis displayed in sub-regions 7402 and 7403, which are positionsdesignated as regions “right” and “left,” at the time of theimplementation in step S16.

In addition, line 13 of FIG. 75 describes that video contents“tvcontents” are reproduced and displayed in main region 7401 (main),and such video contents “tvcontents” are movie data or a TV channel thatare determined in accordance with the description of the script shown inFIG. 76. The description in line 5 of FIG. 76 describes that the scriptshown in FIG. 76 designates a portion of media data of which the ID inthe script shown in FIG. 75 is “tvcontents.”

Next, the flow of the determination process for determining these videocontents “tvcontents” in TV 6 is described using the flowchart of FIG.78.

With reference to FIG. 78, first, when it has been analyzed in TV 6 thatthere is a designation ‘video id=“tvcontents”’ in the drawinginformation describing portion of the script (YES in S2301), the programtable information that is the result of the analysis of the script shownin FIG. 76 is referred to so as to acquire program information thatcorresponds to the present time (S2302).

Lines 6 to 13 of FIG. 76 describes that the program of channel number 4of a TV corresponds to video contents “tvcontents” between 0 o'clock and2 o'clock, and lines 14 to 21 describe that the internet contents whichare designated by the URL described in line 20 correspond to videocontents “tvcontents” between 2 o'clock and 4 o'clock. Therefore, videocontents “tvcontents” are determined to be the television program ofchannel number 4 according to the above description when this time is atime between 0 o'clock and 2 o'clock, and therefore, the type of thisprogram information (TV program) is determined in step S2303, and TV 6receives the corresponding broadcasting waves from broadcasting station5 and carries out a display in a predetermined position in accordancewith the above-described results of the determination (S2304).

Meanwhile, video contents “tvcontents” are determined to be the internetcontents which are designated by the URL described in line 20 of FIG. 76according to the above description when this time is a time between 2o'clock and 4 o'clock, and therefore, the type of this programinformation (internet contents) is determined in step S2303, and TV 6acquires the corresponding contents data from WWW server 4 and carriesout a display in a predetermined position in accordance with theabove-described results of the determination (S2305).

Here, though the portion of program description at and after 4 o'clockis omitted in the script shown in FIG. 76, there is a similardescription for this time, and a similar process is carried out whenthis time is a time other than the above-described time.

As a result of the above-described processing, in TV 6, the media datathat has been acquired from WWW server 4 is reproduced and displayed insub-regions 7402 and 7403, and a TV program or internet contents whichare designated in accordance with a time slot on the basis of theprogram table information that is obtained by analyzing the script shownin FIG. 76 are acquired from broadcasting station 5 or WWW server 4 soas to be reproduced and displayed in main region 7401.

In addition, though in the present embodiment, still image data that hasbeen acquired from a WWW server is displayed in sub-regions 7402 and7403, the description in lines 14 and 15 of FIG. 75 may be converted toa similar description as that of line 13 so as to be combined with thedescription of the script shown in FIG. 76, or description concerningswitching of media data or events which occur as time passes asdescribed in the first and second embodiments may be added, and thereby,broadcasting waves, program related information and the like may beacquired from EPG server 3, WWW server 4 and broadcasting station 5 inaccordance with a time slot so as to be displayed, or the UI componentsthat receive the operation by the user may be displayed so as toimplement the corresponding actions in the case where an operation iscarried out on these UI components by the user.

Furthermore, switching of channels and adjustment of sound volume,brightness and the like of a TV, transition of a link to WEB contentsfor displaying information that interlocks with the program that isreproduced in main region 7401 when a user presses a button that isdisplayed as “detailed information” or the like, recording operation inthe case where TV 6 is provided with a recording function or isconnected to a recording apparatus, transmission of a screen shot whichis recorded from the screen to another user by means of an e-mail or thelike, signal issue to a predetermined dial number in the case where TV 6is interlocked with a telephone function and the like can be cited asthe above-described actions.

The data communication system that is formed so as to include TV 6 whichis a multimedia data processing device according to the presentembodiment has the above-described configuration, and therefore, has anadvantage, primarily for those who prepare content, where multimediacontents data can easily be prepared by combining TV programs andinternet contents.

Here, though a case where a template that is provided by server 2 isprocessed in cellular phone 1 or TV 6 of which the configuration isshown in FIG. 2 as a concrete example is described in the first to thirdembodiments and in the modification of the second embodiment, theprocessing device for processing a template is not limited to a devicelike cellular phone 1, where all the functions are integrally formed.That is to say, the respective functions implemented in cellular phoneor TV 6 may be achieved in devices which are independent from eachother, such as an acquiring device for acquiring a template, like areceiver, a media data inputting device for acquiring an image, like acamera, and a media data outputting device, like a television receiver.In such a case also, data and control signals are mutually sent andreceived by means of communication or wireless communication via adedicated line, reproducing and editing processing of theabove-described script is implemented in the data processing device, theacquired script is processed in the acquiring device, and editing ofmultimedia contents can be carried out by using media data or the likethat has been inputted by the media inputting device. Then, a templatethat has been acquired by the media data outputting device or multimediadata where such a template is edited can be reproduced.

Furthermore, a script implementing method for implementing such a scriptcan be provided as a program. Such a program can be recorded in acomputer readable recording medium, such as a flexible disk, a CD-ROM(Compact Disc-Read Only Memory), a ROM (Read Only Memory), a RAM (RandomAccess Memory) or a memory card, which is used with a computer, so as tobe provided as a program product. Alternatively, a program can beprovided by being recorded in a recording medium, such as a hard disk,which is built into a computer. In addition, a program can be providedby downloading via a network.

A program product that is provided in such a manner is installed in aprogram storing unit, such as a hard disk, and then, is implemented.Here, a program product includes a program itself and a recording mediumin which a program is recorded.

The embodiments which are disclosed herein are illustrative in allrespects, and should not be considered to be limitative. The scope ofthe present invention is defined not by the above description, but bythe claims, and is intended to include meanings which are equivalent tothe claims and all the modifications within the scope.

INDUSTRIAL APPLICABILITY

As described above, multimedia contents can be easily prepared accordingto this invention, and therefore, this invention can be advantageouslyapplied to the preparation of multimedia contents.

The invention claimed is:
 1. A non-transitory computer-readablerecording medium recording multimedia contents data and a data structurethat represents a chronological sequence of operations that areprocessed in a data processing device that includes a reproducing unitfor reproducing predetermined media data and an inputting unit forreceiving an input operation from a user, said non-transitorycomputer-readable recording medium further comprises: a schedule portionincluding a predetermined reproduction time when said media data is tobe reproduced in said reproducing unit and a predetermined input timewhen said input operation is to be received by said input unit: areproduction portion that describes the media data to be reproduced insaid reproducing unit in accordance with said chronological sequence;and an input portion that describes a predetermined editing operationthat is to be input by said inputting unit at said predetermined inputtime in accordance with said chronological sequence, wherein whenexecuted, said data processing device performs steps of processing saidmultimedia contents data so that the predetermined media data isreproduced in said reproducing unit at the predetermined reproductiontime on the basis of said schedule portion, receiving a predeterminedinput operation by said inputting unit at said predetermined input time,and said data processing device carries out an editing process thatcorresponds to said predetermined editing operation on the basis of saidinput portion, and resuming processing of said multimedia contents inaccordance with the schedule portion.
 2. The non-transitorycomputer-readable recording medium according to claim 1, wherein saidediting process is a process for adding media data to be reproduced atsaid predetermined input operation start time in accordance with saidchronological sequence.
 3. The non-transitory computer-readablerecording medium according to claim 2, wherein said process for addingsaid media data is a process for replacing a portion of said media datain said chronological sequence.
 4. The non-transitory computer-readablerecording medium according to claim 2, wherein said process for addingsaid media data is a process of adding media data, that is obtained byreplacing a portion of said media data in said chronological sequence.5. The non-transitory computer-readable recording medium according toclaim 2, wherein said process for adding said media data is a processfor adding media data, that is obtained by replacing a portion of mediadata in said chronological sequence in a predetermined subsequentprocess.
 6. The non-transitory computer-readable recording mediumaccording to claim 1, wherein said data structure that represents thechronological sequence is in accordance with a script language.